[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:37 dist-test-slave-r7ph krb5kdc[7647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:37 dist-test-slave-r7ph krb5kdc[7647](info): set up 2 sockets
Sep 29 17:57:37 dist-test-slave-r7ph krb5kdc[7647](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 17:57:38.683435 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 17:57:38 dist-test-slave-r7ph krb5kdc[7647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 17:57:38.827963 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:34383
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:34383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:38.952613 7660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:38.952993 7660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:38.953073 7660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:38.957711 7660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:38.957861 7660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:38.957904 7660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:38.957985 7660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:38.958026 7660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:38.963873 7660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:34383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:34383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:38.965131 7660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:38.966310 7660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:38.975790 7665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:38.975781 7668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:38.975781 7666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:38.976159 7660 server_base.cc:1034] running on GCE node
I20260929 17:57:38.976796 7660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:38.978011 7660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:38.979295 7660 hybrid_clock.cc:648] HybridClock initialized: now 1790704658979283 us; error 69 us; skew 500 ppm
Sep 29 17:57:38 dist-test-slave-r7ph krb5kdc[7647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704658, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:38.982745 7660 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 17:57:38.983827 7660 webserver.cc:460] Webserver started at http://127.7.118.62:40267/ using document root <none> and password file <none>
I20260929 17:57:38.984462 7660 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:38.984534 7660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:38.984750 7660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:38.987294 7660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "883d535b25764a2398127f21e0fe1e59"
format_stamp: "Formatted at 2026-09-29 17:57:38 on dist-test-slave-r7ph"
server_key: "c94c635c4fc323d4cc17923afa5912ae"
server_key_iv: "643b979e0732e27a9234062f3265d9b6"
server_key_version: "encryptionkey@0"
I20260929 17:57:38.987943 7660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "883d535b25764a2398127f21e0fe1e59"
format_stamp: "Formatted at 2026-09-29 17:57:38 on dist-test-slave-r7ph"
server_key: "c94c635c4fc323d4cc17923afa5912ae"
server_key_iv: "643b979e0732e27a9234062f3265d9b6"
server_key_version: "encryptionkey@0"
I20260929 17:57:38.992662 7660 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 17:57:38.996038 7675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:38.998104 7660 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260929 17:57:38.998543 7660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "883d535b25764a2398127f21e0fe1e59"
format_stamp: "Formatted at 2026-09-29 17:57:38 on dist-test-slave-r7ph"
server_key: "c94c635c4fc323d4cc17923afa5912ae"
server_key_iv: "643b979e0732e27a9234062f3265d9b6"
server_key_version: "encryptionkey@0"
I20260929 17:57:38.998728 7660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:39.014542 7660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:39.018682 7660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:39.018935 7660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:39.027064 7660 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:34383
I20260929 17:57:39.027081 7727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:34383 every 8 connection(s)
I20260929 17:57:39.028159 7660 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:39.031664 7728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:39.036748 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 7660
I20260929 17:57:39.036939 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 17:57:39.038729 7640 external_mini_cluster.cc:1428] Setting key e366497665e909fee63db810d0733884
I20260929 17:57:39.038827 7728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 883d535b25764a2398127f21e0fe1e59: Bootstrap starting.
I20260929 17:57:39.042135 7728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 883d535b25764a2398127f21e0fe1e59: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:39.043251 7728 log.cc:824] T 00000000000000000000000000000000 P 883d535b25764a2398127f21e0fe1e59: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:39.045385 7728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 883d535b25764a2398127f21e0fe1e59: No bootstrap required, opened a new log
I20260929 17:57:39.050386 7728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 883d535b25764a2398127f21e0fe1e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "883d535b25764a2398127f21e0fe1e59" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 34383 } }
I20260929 17:57:39.050719 7728 raft_consensus.cc:374] T 00000000000000000000000000000000 P 883d535b25764a2398127f21e0fe1e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:39.050815 7728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 883d535b25764a2398127f21e0fe1e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 883d535b25764a2398127f21e0fe1e59, State: Initialized, Role: FOLLOWER
I20260929 17:57:39.051354 7728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 883d535b25764a2398127f21e0fe1e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "883d535b25764a2398127f21e0fe1e59" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 34383 } }
I20260929 17:57:39.051522 7728 raft_consensus.cc:388] T 00000000000000000000000000000000 P 883d535b25764a2398127f21e0fe1e59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:39.051620 7728 raft_consensus.cc:482] T 00000000000000000000000000000000 P 883d535b25764a2398127f21e0fe1e59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:39.051743 7728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 883d535b25764a2398127f21e0fe1e59 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:39.052778 7728 raft_consensus.cc:504] T 00000000000000000000000000000000 P 883d535b25764a2398127f21e0fe1e59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "883d535b25764a2398127f21e0fe1e59" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 34383 } }
I20260929 17:57:39.053198 7728 leader_election.cc:302] T 00000000000000000000000000000000 P 883d535b25764a2398127f21e0fe1e59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 883d535b25764a2398127f21e0fe1e59; no voters:
I20260929 17:57:39.053588 7728 leader_election.cc:288] T 00000000000000000000000000000000 P 883d535b25764a2398127f21e0fe1e59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:39.053742 7733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 883d535b25764a2398127f21e0fe1e59 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:39.054082 7733 raft_consensus.cc:686] T 00000000000000000000000000000000 P 883d535b25764a2398127f21e0fe1e59 [term 1 LEADER]: Becoming Leader. State: Replica: 883d535b25764a2398127f21e0fe1e59, State: Running, Role: LEADER
I20260929 17:57:39.054544 7733 consensus_queue.cc:237] T 00000000000000000000000000000000 P 883d535b25764a2398127f21e0fe1e59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "883d535b25764a2398127f21e0fe1e59" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 34383 } }
I20260929 17:57:39.055110 7728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 883d535b25764a2398127f21e0fe1e59 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:39.056697 7734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 883d535b25764a2398127f21e0fe1e59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "883d535b25764a2398127f21e0fe1e59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "883d535b25764a2398127f21e0fe1e59" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 34383 } } }
I20260929 17:57:39.056854 7734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 883d535b25764a2398127f21e0fe1e59 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:39.057336 7735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 883d535b25764a2398127f21e0fe1e59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 883d535b25764a2398127f21e0fe1e59. Latest consensus state: current_term: 1 leader_uuid: "883d535b25764a2398127f21e0fe1e59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "883d535b25764a2398127f21e0fe1e59" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 34383 } } }
I20260929 17:57:39.058173 7735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 883d535b25764a2398127f21e0fe1e59 [sys.catalog]: This master's current role is: LEADER
W20260929 17:57:39.061110 7748 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 883d535b25764a2398127f21e0fe1e59: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:57:39.061219 7748 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:57:39.061295 7747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:39.062419 7747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 17:57:39 dist-test-slave-r7ph krb5kdc[7647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:57:39.070428 7747 catalog_manager.cc:1347] Generated new cluster ID: 333dc37fb6c74123a1fc06c598b2eff5
I20260929 17:57:39.070528 7747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:39.072156 7730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:39.042646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59250 (local address 127.7.118.62:34383)
0929 17:57:39.043201 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:39.043222 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:39.044549 (+ 1327us) server_negotiation.cc:407] Connection header received
0929 17:57:39.048487 (+ 3938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:39.048517 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:39.049092 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:39.049606 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:39.053050 (+ 3444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:39.054597 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:39.057045 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:39.057955 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:39.058161 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:39.064070 (+ 5909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:39.064078 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:39.064099 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:39.064143 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:39.067258 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:39.068381 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:39.068389 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:39.068395 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:39.068607 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:39.069153 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:39.069157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:39.069158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:39.069749 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:39.069975 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:39.070826 (+ 851us) server_negotiation.cc:299] Negotiation successful
0929 17:57:39.071169 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":157,"threads_started":1}
I20260929 17:57:39.079312 7747 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:39.080415 7747 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:39.088479 7747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 883d535b25764a2398127f21e0fe1e59: Generated new TSK 0
I20260929 17:57:39.089382 7747 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.118.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
I20260929 17:57:39.127853 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:0
--local_ip_for_outbound_sockets=127.7.118.1
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:34383
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:39.261857 7754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:39.262217 7754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:39.262358 7754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:39.267321 7754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:39.267406 7754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:39.267473 7754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 17:57:39.273561 7754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:34383
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:39.274735 7754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:39.275899 7754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:39.287264 7762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:39.287186 7760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:39.287186 7759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:39.287343 7754 server_base.cc:1034] running on GCE node
I20260929 17:57:39.288288 7754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:39.289088 7754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:39.290428 7754 hybrid_clock.cc:648] HybridClock initialized: now 1790704659290342 us; error 36 us; skew 500 ppm
Sep 29 17:57:39 dist-test-slave-r7ph krb5kdc[7647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704659, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:39.294112 7754 init.cc:377] Logged in from keytab as kudu/127.7.118.1@KRBTEST.COM (short username kudu)
I20260929 17:57:39.295603 7754 webserver.cc:460] Webserver started at http://127.7.118.1:38765/ using document root <none> and password file <none>
I20260929 17:57:39.296279 7754 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:39.296358 7754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:39.296602 7754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:39.299058 7754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-0/data/instance:
uuid: "d990792bdd6a4471a6dd2f478971ad06"
format_stamp: "Formatted at 2026-09-29 17:57:39 on dist-test-slave-r7ph"
server_key: "edb9a9726395fbdbf181361bfea7043a"
server_key_iv: "f30a74a7b7b433d18c72696de83572c8"
server_key_version: "encryptionkey@0"
I20260929 17:57:39.299719 7754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance:
uuid: "d990792bdd6a4471a6dd2f478971ad06"
format_stamp: "Formatted at 2026-09-29 17:57:39 on dist-test-slave-r7ph"
server_key: "edb9a9726395fbdbf181361bfea7043a"
server_key_iv: "f30a74a7b7b433d18c72696de83572c8"
server_key_version: "encryptionkey@0"
I20260929 17:57:39.304648 7754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 17:57:39.307909 7769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:39.309849 7754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 17:57:39.310030 7754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "d990792bdd6a4471a6dd2f478971ad06"
format_stamp: "Formatted at 2026-09-29 17:57:39 on dist-test-slave-r7ph"
server_key: "edb9a9726395fbdbf181361bfea7043a"
server_key_iv: "f30a74a7b7b433d18c72696de83572c8"
server_key_version: "encryptionkey@0"
I20260929 17:57:39.310228 7754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:39.329532 7754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:39.333662 7754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:39.333914 7754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:39.334808 7754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:57:39.335772 7754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:57:39.335824 7754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:39.335866 7754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:57:39.335925 7754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:39.345863 7754 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:39165
I20260929 17:57:39.346014 7882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:39165 every 8 connection(s)
I20260929 17:57:39.346933 7754 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 17:57:39.356617 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 7754
I20260929 17:57:39.356765 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance
I20260929 17:57:39.357136 7640 external_mini_cluster.cc:1428] Setting key c793835849bfd1f1dbab1c31d48d2e10
Sep 29 17:57:39 dist-test-slave-r7ph krb5kdc[7647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704659, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:57:39.364722 7730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:39.349133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:47965 (local address 127.7.118.62:34383)
0929 17:57:39.349295 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:39.349299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:39.350463 (+ 1164us) server_negotiation.cc:407] Connection header received
0929 17:57:39.352132 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:39.352136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:39.352200 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:39.352317 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:39.353683 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:39.354229 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:39.355684 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:39.355977 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:39.356104 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:39.359763 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:39.359767 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:39.359769 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:39.359796 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:39.361953 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:39.362714 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:39.362721 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:39.362725 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:39.362832 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:39.363244 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:39.363250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:39.363255 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:39.363557 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:39.363676 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:39.364293 (+ 617us) server_negotiation.cc:299] Negotiation successful
0929 17:57:39.364455 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 17:57:39.365578 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:39.349601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:34383 (local address 127.7.118.1:47965)
0929 17:57:39.350150 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:39.350205 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:39.351842 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:39.352525 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:39.352535 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:39.352948 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:39.353503 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:39.353518 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:39.354430 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:39.354433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:39.355523 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:39.355536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:39.356172 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:39.356176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:39.356424 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:39.357170 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:39.357227 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:39.359537 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:39.362139 (+ 2602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:39.362146 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:39.362163 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:39.362593 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:39.362938 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:39.362940 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:39.362941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:39.363126 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:39.363689 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:39.363696 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:39.364006 (+ 310us) client_negotiation.cc:769] Sending connection context
0929 17:57:39.364185 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 17:57:39.364504 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":136,"threads_started":1}
I20260929 17:57:39.367103 7883 heartbeater.cc:347] Connected to a master server at 127.7.118.62:34383
I20260929 17:57:39.367393 7883 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.7.118.2@KRBTEST.COM; defaulting to no policy
I20260929 17:57:39.368034 7883 heartbeater.cc:511] Master 127.7.118.62:34383 requested a full tablet report, sending...
I20260929 17:57:39.370383 7692 ts_manager.cc:194] Registered new tserver with Master: d990792bdd6a4471a6dd2f478971ad06 (127.7.118.1:39165)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.2@KRBTEST.COM" created.
I20260929 17:57:39.372793 7692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.1@KRBTEST.COM'} at 127.7.118.1:47965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
I20260929 17:57:39.399389 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:0
--local_ip_for_outbound_sockets=127.7.118.2
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:34383
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:39.533308 7888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:39.533643 7888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:39.533689 7888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:39.538039 7888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:39.538125 7888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:39.538189 7888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 17:57:39.544318 7888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:34383
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:39.545440 7888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:39.546597 7888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:39.557340 7894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:39.557340 7896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:39.557356 7893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:39.557837 7888 server_base.cc:1034] running on GCE node
I20260929 17:57:39.558322 7888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:39.559094 7888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:39.560294 7888 hybrid_clock.cc:648] HybridClock initialized: now 1790704659560260 us; error 52 us; skew 500 ppm
Sep 29 17:57:39 dist-test-slave-r7ph krb5kdc[7647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704659, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:39.563970 7888 init.cc:377] Logged in from keytab as kudu/127.7.118.2@KRBTEST.COM (short username kudu)
I20260929 17:57:39.565038 7888 webserver.cc:460] Webserver started at http://127.7.118.2:42331/ using document root <none> and password file <none>
I20260929 17:57:39.565656 7888 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:39.565726 7888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:39.565958 7888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:39.568302 7888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-1/data/instance:
uuid: "01e468f59c1045c9b1a12ca83fbedd25"
format_stamp: "Formatted at 2026-09-29 17:57:39 on dist-test-slave-r7ph"
server_key: "52edba3c660d7ea1b9a6047dfc3f7a68"
server_key_iv: "a55e89833228512f529da8265cb4c43e"
server_key_version: "encryptionkey@0"
I20260929 17:57:39.568951 7888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance:
uuid: "01e468f59c1045c9b1a12ca83fbedd25"
format_stamp: "Formatted at 2026-09-29 17:57:39 on dist-test-slave-r7ph"
server_key: "52edba3c660d7ea1b9a6047dfc3f7a68"
server_key_iv: "a55e89833228512f529da8265cb4c43e"
server_key_version: "encryptionkey@0"
I20260929 17:57:39.573513 7888 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 17:57:39.576654 7903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:39.578346 7888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:57:39.578526 7888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "01e468f59c1045c9b1a12ca83fbedd25"
format_stamp: "Formatted at 2026-09-29 17:57:39 on dist-test-slave-r7ph"
server_key: "52edba3c660d7ea1b9a6047dfc3f7a68"
server_key_iv: "a55e89833228512f529da8265cb4c43e"
server_key_version: "encryptionkey@0"
I20260929 17:57:39.578670 7888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:39.594772 7888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:39.599190 7888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:39.599454 7888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:39.600185 7888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:57:39.601231 7888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:57:39.601284 7888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:39.601364 7888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:57:39.601400 7888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:39.612790 7888 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:44615
I20260929 17:57:39.612825 8016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:44615 every 8 connection(s)
I20260929 17:57:39.613835 7888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 17:57:39.617559 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 7888
I20260929 17:57:39.617712 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance
I20260929 17:57:39.618036 7640 external_mini_cluster.cc:1428] Setting key 78c790164c27548b938c2e57d6155042
Sep 29 17:57:39 dist-test-slave-r7ph krb5kdc[7647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704659, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.118.3@KRBTEST.COM; defaulting to no policy
I20260929 17:57:39.634286 7730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:39.616376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:43745 (local address 127.7.118.62:34383)
0929 17:57:39.616523 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:39.616528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:39.617776 (+ 1248us) server_negotiation.cc:407] Connection header received
0929 17:57:39.619858 (+ 2082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:39.619862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:39.619919 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:39.619998 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:39.622212 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:39.623169 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:39.624862 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:39.625175 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:39.625297 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:39.629538 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:39.629542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:39.629545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:39.629578 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:39.631717 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:39.632484 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:39.632492 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:39.632494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:39.632580 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:39.633014 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:39.633018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:39.633020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:39.633236 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:39.633357 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:39.633945 (+ 588us) server_negotiation.cc:299] Negotiation successful
0929 17:57:39.634087 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.3@KRBTEST.COM" created.
I20260929 17:57:39.635308 8019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:39.616837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:34383 (local address 127.7.118.2:43745)
0929 17:57:39.617544 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:39.617626 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:39.619606 (+ 1980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:39.620285 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:39.620298 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:39.621008 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:39.621936 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:39.621970 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:39.623382 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:39.623386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:39.624647 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:39.624664 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:39.625420 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:39.625427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:39.625728 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:39.626703 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:39.626724 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:39.629245 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:39.631901 (+ 2656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:39.631909 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:39.631929 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:39.632337 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:39.632686 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:39.632690 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:39.632692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:39.632898 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:39.633354 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:39.633362 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:39.633670 (+ 308us) client_negotiation.cc:769] Sending connection context
0929 17:57:39.633904 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 17:57:39.634335 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":129,"threads_started":1}
I20260929 17:57:39.636868 8017 heartbeater.cc:347] Connected to a master server at 127.7.118.62:34383
I20260929 17:57:39.637209 8017 heartbeater.cc:464] Registering TS with master...
I20260929 17:57:39.637799 8017 heartbeater.cc:511] Master 127.7.118.62:34383 requested a full tablet report, sending...
I20260929 17:57:39.639217 7692 ts_manager.cc:194] Registered new tserver with Master: 01e468f59c1045c9b1a12ca83fbedd25 (127.7.118.2:44615)
I20260929 17:57:39.640236 7692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.2@KRBTEST.COM'} at 127.7.118.2:43745
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
I20260929 17:57:39.658313 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:0
--local_ip_for_outbound_sockets=127.7.118.3
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:34383
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:39.784559 8022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:39.784902 8022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:39.784993 8022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:39.789860 8022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:39.789948 8022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:39.790053 8022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 17:57:39.795826 8022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:34383
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:39.797049 8022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:39.798189 8022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:39.808780 8030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:39.808748 8028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:39.808826 8022 server_base.cc:1034] running on GCE node
W20260929 17:57:39.808758 8027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:39.809569 8022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:39.810369 8022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:39.811589 8022 hybrid_clock.cc:648] HybridClock initialized: now 1790704659811542 us; error 57 us; skew 500 ppm
Sep 29 17:57:39 dist-test-slave-r7ph krb5kdc[7647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704659, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:39.815191 8022 init.cc:377] Logged in from keytab as kudu/127.7.118.3@KRBTEST.COM (short username kudu)
I20260929 17:57:39.816247 8022 webserver.cc:460] Webserver started at http://127.7.118.3:36269/ using document root <none> and password file <none>
I20260929 17:57:39.816918 8022 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:39.816988 8022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:39.817214 8022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:39.819669 8022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-2/data/instance:
uuid: "648ae676ecc64a5aa8884f58f29c006e"
format_stamp: "Formatted at 2026-09-29 17:57:39 on dist-test-slave-r7ph"
server_key: "e810a6d14982ab2cfc10cf990745bd35"
server_key_iv: "72557dae7b5746a3a9bb580aecd7bd83"
server_key_version: "encryptionkey@0"
I20260929 17:57:39.820307 8022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance:
uuid: "648ae676ecc64a5aa8884f58f29c006e"
format_stamp: "Formatted at 2026-09-29 17:57:39 on dist-test-slave-r7ph"
server_key: "e810a6d14982ab2cfc10cf990745bd35"
server_key_iv: "72557dae7b5746a3a9bb580aecd7bd83"
server_key_version: "encryptionkey@0"
I20260929 17:57:39.824705 8022 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:57:39.827773 8037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:39.829389 8022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 17:57:39.829535 8022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "648ae676ecc64a5aa8884f58f29c006e"
format_stamp: "Formatted at 2026-09-29 17:57:39 on dist-test-slave-r7ph"
server_key: "e810a6d14982ab2cfc10cf990745bd35"
server_key_iv: "72557dae7b5746a3a9bb580aecd7bd83"
server_key_version: "encryptionkey@0"
I20260929 17:57:39.829655 8022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:39.857954 8022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:39.861843 8022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:39.862028 8022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:39.862684 8022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:57:39.863605 8022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:57:39.863656 8022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:39.863703 8022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:57:39.863719 8022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:39.873430 8022 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:36565
I20260929 17:57:39.873450 8150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:36565 every 8 connection(s)
I20260929 17:57:39.874572 8022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 17:57:39.875950 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 8022
I20260929 17:57:39.876096 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance
I20260929 17:57:39.876408 7640 external_mini_cluster.cc:1428] Setting key c23a8cfb63a88106d63ae5b32d6f971f
Sep 29 17:57:39 dist-test-slave-r7ph krb5kdc[7647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704659, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:57:39.890642 7730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:39.876710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:40263 (local address 127.7.118.62:34383)
0929 17:57:39.876861 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:39.876867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:39.877870 (+ 1003us) server_negotiation.cc:407] Connection header received
0929 17:57:39.879349 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:39.879352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:39.879404 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:39.879474 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:39.880715 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:39.881266 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:39.882637 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:39.882931 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:39.883027 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:39.886138 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:39.886142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:39.886144 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:39.886174 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:39.887818 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:39.888493 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:39.888497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:39.888498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:39.888589 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:39.889023 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:39.889027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:39.889028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:39.889249 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:39.889338 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:39.890226 (+ 888us) server_negotiation.cc:299] Negotiation successful
0929 17:57:39.890494 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 17:57:39.891709 8153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:39.877190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:34383 (local address 127.7.118.3:40263)
0929 17:57:39.877696 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:39.877754 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:39.879136 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:39.879652 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:39.879659 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:39.880041 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:39.880554 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:39.880569 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:39.881400 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:39.881402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:39.882506 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:39.882517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:39.883151 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:39.883154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:39.883358 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:39.884097 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:39.884114 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:39.885845 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:39.888015 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:39.888021 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:39.888038 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:39.888374 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:39.888722 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:39.888725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:39.888726 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:39.888904 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:39.889399 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:39.889406 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:39.889775 (+ 369us) client_negotiation.cc:769] Sending connection context
0929 17:57:39.890085 (+ 310us) client_negotiation.cc:240] Negotiation successful
0929 17:57:39.890581 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":154,"threads_started":1}
I20260929 17:57:39.893229 8151 heartbeater.cc:347] Connected to a master server at 127.7.118.62:34383
I20260929 17:57:39.893559 8151 heartbeater.cc:464] Registering TS with master...
I20260929 17:57:39.894361 8151 heartbeater.cc:511] Master 127.7.118.62:34383 requested a full tablet report, sending...
I20260929 17:57:39.895789 7692 ts_manager.cc:194] Registered new tserver with Master: 648ae676ecc64a5aa8884f58f29c006e (127.7.118.3:36565)
I20260929 17:57:39.896548 7692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.3@KRBTEST.COM'} at 127.7.118.3:40263
I20260929 17:57:39.904621 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 17:57:39 dist-test-slave-r7ph krb5kdc[7647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704659, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 17:57:39 dist-test-slave-r7ph krb5kdc[7647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704659, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.118.1@KRBTEST.COM
I20260929 17:57:39.934921 8158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:39.923299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47416 (local address 127.7.118.1:39165)
0929 17:57:39.923613 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:39.923624 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:39.923654 (+ 30us) server_negotiation.cc:407] Connection header received
0929 17:57:39.923841 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:39.923853 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:39.924092 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:39.924257 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:39.925030 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:39.925820 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:39.926802 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:39.927219 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:39.927353 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:39.930215 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:39.930228 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:39.930311 (+ 83us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:39.930358 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:39.932280 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:39.932919 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:39.932925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:39.932926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:39.933027 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:39.933393 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:39.933397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:39.933399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:39.933736 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:39.933998 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:39.934379 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 17:57:39.934635 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":66,"threads_started":1}
W20260929 17:57:39.936173 7797 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47416
Sep 29 17:57:39 dist-test-slave-r7ph krb5kdc[7647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 17:57:39 dist-test-slave-r7ph krb5kdc[7647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.1@KRBTEST.COM
I20260929 17:57:39.965725 8158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:39.955452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47426 (local address 127.7.118.1:39165)
0929 17:57:39.955641 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:39.955645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:39.955663 (+ 18us) server_negotiation.cc:407] Connection header received
0929 17:57:39.955828 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:39.955831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:39.955876 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:39.956008 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:39.956999 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:39.957539 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:39.958645 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:39.958899 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:39.958999 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:39.961971 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:39.961975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:39.961977 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:39.962012 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:39.963670 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:39.964238 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:39.964241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:39.964243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:39.964335 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:39.964714 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:39.964717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:39.964718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:39.964908 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:39.965059 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:39.965317 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 17:57:39.965484 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 17:57:39.967130 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 7754
I20260929 17:57:39.972638 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 7888
I20260929 17:57:39.978870 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 8022
I20260929 17:57:39.984671 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 7660
2026-09-29T17:57:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2411 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:40 dist-test-slave-r7ph krb5kdc[8165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:40 dist-test-slave-r7ph krb5kdc[8165](info): set up 2 sockets
Sep 29 17:57:40 dist-test-slave-r7ph krb5kdc[8165](info): commencing operation
krb5kdc: starting...
W20260929 17:57:41.034157 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 17:57:41 dist-test-slave-r7ph krb5kdc[8165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 17:57:41.166530 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:39613
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:45203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:39613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:41.283416 8178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:41.283779 8178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:41.283870 8178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:41.288441 8178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:41.288551 8178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:41.288589 8178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:41.288630 8178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:41.288661 8178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:41.294480 8178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:45203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:39613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:39613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:41.295707 8178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:41.296865 8178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:41.307116 8186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:41.307119 8183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:41.307119 8184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:41.307485 8178 server_base.cc:1034] running on GCE node
I20260929 17:57:41.308216 8178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:41.309526 8178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:41.310748 8178 hybrid_clock.cc:648] HybridClock initialized: now 1790704661310715 us; error 52 us; skew 500 ppm
Sep 29 17:57:41 dist-test-slave-r7ph krb5kdc[8165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704661, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:41.313969 8178 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 17:57:41.315115 8178 webserver.cc:460] Webserver started at http://127.7.118.62:38503/ using document root <none> and password file <none>
I20260929 17:57:41.315755 8178 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:41.315863 8178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:41.316119 8178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:41.318442 8178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "23251b7439da414c9a19ac23ab1aee60"
format_stamp: "Formatted at 2026-09-29 17:57:41 on dist-test-slave-r7ph"
server_key: "dd8ee6d9461ff96f3ddcd98b12d3bed1"
server_key_iv: "68da215745ae4d5c55cc77a822a91641"
server_key_version: "encryptionkey@0"
I20260929 17:57:41.319092 8178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "23251b7439da414c9a19ac23ab1aee60"
format_stamp: "Formatted at 2026-09-29 17:57:41 on dist-test-slave-r7ph"
server_key: "dd8ee6d9461ff96f3ddcd98b12d3bed1"
server_key_iv: "68da215745ae4d5c55cc77a822a91641"
server_key_version: "encryptionkey@0"
I20260929 17:57:41.323720 8178 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 17:57:41.326964 8193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:41.328570 8178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:57:41.328768 8178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "23251b7439da414c9a19ac23ab1aee60"
format_stamp: "Formatted at 2026-09-29 17:57:41 on dist-test-slave-r7ph"
server_key: "dd8ee6d9461ff96f3ddcd98b12d3bed1"
server_key_iv: "68da215745ae4d5c55cc77a822a91641"
server_key_version: "encryptionkey@0"
I20260929 17:57:41.329027 8178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:41.347118 8178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:41.350751 8178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:41.351014 8178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:41.358949 8178 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:39613
I20260929 17:57:41.358963 8245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:39613 every 8 connection(s)
I20260929 17:57:41.360008 8178 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:41.363268 8246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:41.364018 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 8178
I20260929 17:57:41.364174 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 17:57:41.364470 7640 external_mini_cluster.cc:1428] Setting key f7a4ccf36c35d34517f6f3a138f994fb
I20260929 17:57:41.371254 8246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23251b7439da414c9a19ac23ab1aee60: Bootstrap starting.
I20260929 17:57:41.373992 8246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23251b7439da414c9a19ac23ab1aee60: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:41.374878 8246 log.cc:824] T 00000000000000000000000000000000 P 23251b7439da414c9a19ac23ab1aee60: Log is configured to *not* fsync() on all Append() calls
Sep 29 17:57:41 dist-test-slave-r7ph krb5kdc[8165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:57:41.377367 8246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23251b7439da414c9a19ac23ab1aee60: No bootstrap required, opened a new log
I20260929 17:57:41.381547 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:41.366111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40668 (local address 127.7.118.62:39613)
0929 17:57:41.366657 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:41.366680 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:41.366730 (+ 50us) server_negotiation.cc:407] Connection header received
0929 17:57:41.367665 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:41.367693 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:41.368166 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:41.368603 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:41.369383 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:41.370943 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:41.372067 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:41.372808 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:41.372983 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:41.376199 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:41.376206 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:41.376222 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:41.376252 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:41.378231 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:41.378880 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:41.378888 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:41.378896 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:41.379039 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:41.379382 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:41.379384 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:41.379385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:41.379811 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:41.379974 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:41.380319 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 17:57:41.380691 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":184,"threads_started":1}
I20260929 17:57:41.381837 8246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23251b7439da414c9a19ac23ab1aee60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23251b7439da414c9a19ac23ab1aee60" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 39613 } }
I20260929 17:57:41.382102 8246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23251b7439da414c9a19ac23ab1aee60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:41.382186 8246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23251b7439da414c9a19ac23ab1aee60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23251b7439da414c9a19ac23ab1aee60, State: Initialized, Role: FOLLOWER
I20260929 17:57:41.382748 8246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23251b7439da414c9a19ac23ab1aee60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23251b7439da414c9a19ac23ab1aee60" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 39613 } }
I20260929 17:57:41.382932 8246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23251b7439da414c9a19ac23ab1aee60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:41.383024 8246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23251b7439da414c9a19ac23ab1aee60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:41.383128 8246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23251b7439da414c9a19ac23ab1aee60 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:41.384160 8246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23251b7439da414c9a19ac23ab1aee60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23251b7439da414c9a19ac23ab1aee60" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 39613 } }
I20260929 17:57:41.384543 8246 leader_election.cc:302] T 00000000000000000000000000000000 P 23251b7439da414c9a19ac23ab1aee60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23251b7439da414c9a19ac23ab1aee60; no voters:
I20260929 17:57:41.384932 8246 leader_election.cc:288] T 00000000000000000000000000000000 P 23251b7439da414c9a19ac23ab1aee60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:41.385064 8251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23251b7439da414c9a19ac23ab1aee60 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:41.385354 8251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23251b7439da414c9a19ac23ab1aee60 [term 1 LEADER]: Becoming Leader. State: Replica: 23251b7439da414c9a19ac23ab1aee60, State: Running, Role: LEADER
I20260929 17:57:41.385716 8251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23251b7439da414c9a19ac23ab1aee60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23251b7439da414c9a19ac23ab1aee60" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 39613 } }
I20260929 17:57:41.386418 8246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23251b7439da414c9a19ac23ab1aee60 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:41.387679 8252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23251b7439da414c9a19ac23ab1aee60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23251b7439da414c9a19ac23ab1aee60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23251b7439da414c9a19ac23ab1aee60" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 39613 } } }
I20260929 17:57:41.387692 8253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23251b7439da414c9a19ac23ab1aee60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23251b7439da414c9a19ac23ab1aee60. Latest consensus state: current_term: 1 leader_uuid: "23251b7439da414c9a19ac23ab1aee60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23251b7439da414c9a19ac23ab1aee60" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 39613 } } }
I20260929 17:57:41.387847 8253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23251b7439da414c9a19ac23ab1aee60 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:41.387847 8252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23251b7439da414c9a19ac23ab1aee60 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:41.388289 8260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:41.391916 8260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:41.399010 8260 catalog_manager.cc:1347] Generated new cluster ID: d34913e122ab4c129c591b5f970dce7e
I20260929 17:57:41.399106 8260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:41.411242 8260 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:41.412575 8260 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:41.420706 8260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23251b7439da414c9a19ac23ab1aee60: Generated new TSK 0
I20260929 17:57:41.421533 8260 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.118.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
I20260929 17:57:41.458196 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:0
--local_ip_for_outbound_sockets=127.7.118.1
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:39613
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:45203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:41.595328 8272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:41.595659 8272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:41.595708 8272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:41.600781 8272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:41.600903 8272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:41.600972 8272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 17:57:41.607643 8272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:45203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:39613
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:41.608955 8272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:41.610472 8272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:41.622617 8280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:41.622859 8277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:41.622617 8278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:41.623100 8272 server_base.cc:1034] running on GCE node
I20260929 17:57:41.623605 8272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:41.624332 8272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:41.625663 8272 hybrid_clock.cc:648] HybridClock initialized: now 1790704661625600 us; error 78 us; skew 500 ppm
Sep 29 17:57:41 dist-test-slave-r7ph krb5kdc[8165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704661, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:41.635807 8272 init.cc:377] Logged in from keytab as kudu/127.7.118.1@KRBTEST.COM (short username kudu)
I20260929 17:57:41.637077 8272 webserver.cc:460] Webserver started at http://127.7.118.1:39335/ using document root <none> and password file <none>
I20260929 17:57:41.637771 8272 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:41.637879 8272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:41.638146 8272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:41.640605 8272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-0/data/instance:
uuid: "58db145bc60f4dc5a4e373df3f138f5b"
format_stamp: "Formatted at 2026-09-29 17:57:41 on dist-test-slave-r7ph"
server_key: "e695833b1d6a53ade4a566425683d317"
server_key_iv: "42fabb598d3a0e5d5aac2aaa7c865e83"
server_key_version: "encryptionkey@0"
I20260929 17:57:41.641283 8272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance:
uuid: "58db145bc60f4dc5a4e373df3f138f5b"
format_stamp: "Formatted at 2026-09-29 17:57:41 on dist-test-slave-r7ph"
server_key: "e695833b1d6a53ade4a566425683d317"
server_key_iv: "42fabb598d3a0e5d5aac2aaa7c865e83"
server_key_version: "encryptionkey@0"
I20260929 17:57:41.646226 8272 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260929 17:57:41.649763 8287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:41.651634 8272 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260929 17:57:41.652019 8272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "58db145bc60f4dc5a4e373df3f138f5b"
format_stamp: "Formatted at 2026-09-29 17:57:41 on dist-test-slave-r7ph"
server_key: "e695833b1d6a53ade4a566425683d317"
server_key_iv: "42fabb598d3a0e5d5aac2aaa7c865e83"
server_key_version: "encryptionkey@0"
I20260929 17:57:41.652170 8272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:41.691476 8272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:41.695585 8272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:41.695767 8272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:41.696370 8272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:57:41.697347 8272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:57:41.697397 8272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:41.697440 8272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:57:41.697507 8272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:41.707516 8272 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:45633
I20260929 17:57:41.707535 8400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:45633 every 8 connection(s)
I20260929 17:57:41.708555 8272 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 17:57:41.710362 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 8272
I20260929 17:57:41.710496 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance
I20260929 17:57:41.710807 7640 external_mini_cluster.cc:1428] Setting key ccbfa91137407987ce8f4c687ca9f93d
WARNING: no policy specified for kudu/127.7.118.2@KRBTEST.COM; defaulting to no policy
Sep 29 17:57:41 dist-test-slave-r7ph krb5kdc[8165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704661, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.2@KRBTEST.COM" created.
I20260929 17:57:41.725611 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:41.710845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:43629 (local address 127.7.118.62:39613)
0929 17:57:41.711053 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:41.711057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:41.712205 (+ 1148us) server_negotiation.cc:407] Connection header received
0929 17:57:41.713940 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:41.713944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:41.714005 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:41.714091 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:41.715494 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:41.716091 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:41.717350 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:41.717600 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:41.717721 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:41.720702 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:41.720705 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:41.720708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:41.720738 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:41.722442 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:41.723217 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:41.723232 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:41.723237 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:41.723367 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:41.723954 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:41.723958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:41.723959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:41.724209 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:41.724299 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:41.725257 (+ 958us) server_negotiation.cc:299] Negotiation successful
0929 17:57:41.725425 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 17:57:41.726718 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:41.711480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:39613 (local address 127.7.118.1:43629)
0929 17:57:41.712044 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:41.712109 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:41.713572 (+ 1463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:41.714355 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:41.714363 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:41.714766 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:41.715292 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:41.715305 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:41.716231 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:41.716241 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:41.717202 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:41.717213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:41.717776 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:41.717780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:41.717976 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:41.718662 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:41.718680 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:41.720470 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:41.722635 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:41.722644 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:41.722680 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:41.723093 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:41.723513 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:41.723516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:41.723518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:41.723794 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:41.724338 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:41.724349 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:41.724808 (+ 459us) client_negotiation.cc:769] Sending connection context
0929 17:57:41.725208 (+ 400us) client_negotiation.cc:240] Negotiation successful
0929 17:57:41.725690 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":181,"threads_started":1}
I20260929 17:57:41.728246 8401 heartbeater.cc:347] Connected to a master server at 127.7.118.62:39613
I20260929 17:57:41.728556 8401 heartbeater.cc:464] Registering TS with master...
I20260929 17:57:41.729175 8401 heartbeater.cc:511] Master 127.7.118.62:39613 requested a full tablet report, sending...
I20260929 17:57:41.731014 8209 ts_manager.cc:194] Registered new tserver with Master: 58db145bc60f4dc5a4e373df3f138f5b (127.7.118.1:45633)
I20260929 17:57:41.732268 8209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.1@KRBTEST.COM'} at 127.7.118.1:43629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
I20260929 17:57:41.742122 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:0
--local_ip_for_outbound_sockets=127.7.118.2
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:39613
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:45203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:41.864236 8406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:41.864542 8406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:41.864589 8406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:41.868775 8406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:41.868857 8406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:41.868923 8406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 17:57:41.874503 8406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:45203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:39613
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:41.875582 8406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:41.876678 8406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:41.886924 8412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:41.886904 8411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:41.886935 8414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:41.887686 8406 server_base.cc:1034] running on GCE node
I20260929 17:57:41.888172 8406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:41.888973 8406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:41.890182 8406 hybrid_clock.cc:648] HybridClock initialized: now 1790704661890159 us; error 38 us; skew 500 ppm
Sep 29 17:57:41 dist-test-slave-r7ph krb5kdc[8165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704661, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:41.893430 8406 init.cc:377] Logged in from keytab as kudu/127.7.118.2@KRBTEST.COM (short username kudu)
I20260929 17:57:41.894593 8406 webserver.cc:460] Webserver started at http://127.7.118.2:40911/ using document root <none> and password file <none>
I20260929 17:57:41.895246 8406 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:41.895315 8406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:41.895514 8406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:41.897759 8406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-1/data/instance:
uuid: "51ce51bd45df46c59713ae4ae56fd0e0"
format_stamp: "Formatted at 2026-09-29 17:57:41 on dist-test-slave-r7ph"
server_key: "a7273ff65bafc087eb3666f6399b8511"
server_key_iv: "ca676c82d48e9ad5e383fc16c571bd94"
server_key_version: "encryptionkey@0"
I20260929 17:57:41.898445 8406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance:
uuid: "51ce51bd45df46c59713ae4ae56fd0e0"
format_stamp: "Formatted at 2026-09-29 17:57:41 on dist-test-slave-r7ph"
server_key: "a7273ff65bafc087eb3666f6399b8511"
server_key_iv: "ca676c82d48e9ad5e383fc16c571bd94"
server_key_version: "encryptionkey@0"
I20260929 17:57:41.902680 8406 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 17:57:41.905431 8421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:41.906977 8406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:57:41.907153 8406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "51ce51bd45df46c59713ae4ae56fd0e0"
format_stamp: "Formatted at 2026-09-29 17:57:41 on dist-test-slave-r7ph"
server_key: "a7273ff65bafc087eb3666f6399b8511"
server_key_iv: "ca676c82d48e9ad5e383fc16c571bd94"
server_key_version: "encryptionkey@0"
I20260929 17:57:41.907282 8406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:41.933049 8406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:41.937011 8406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:41.937247 8406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:41.937891 8406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:57:41.938848 8406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:57:41.938920 8406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:41.938961 8406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:57:41.938998 8406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:41.948385 8406 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:43109
I20260929 17:57:41.948482 8534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:43109 every 8 connection(s)
I20260929 17:57:41.949450 8406 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 17:57:41.949775 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 8406
I20260929 17:57:41.949910 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance
I20260929 17:57:41.950212 7640 external_mini_cluster.cc:1428] Setting key 8d0d15dc7185eaadc11c4cdc13b1af3b
WARNING: no policy specified for kudu/127.7.118.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.3@KRBTEST.COM" created.
Sep 29 17:57:41 dist-test-slave-r7ph krb5kdc[8165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704661, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:57:41.969784 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:41.952389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:44559 (local address 127.7.118.62:39613)
0929 17:57:41.952522 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:41.952526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:41.953847 (+ 1321us) server_negotiation.cc:407] Connection header received
0929 17:57:41.955578 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:41.955582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:41.955648 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:41.955775 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:41.957392 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:41.958006 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:41.959494 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:41.959746 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:41.959879 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:41.965501 (+ 5622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:41.965504 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:41.965506 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:41.965540 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:41.967237 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:41.967922 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:41.967926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:41.967928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:41.968020 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:41.968468 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:41.968472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:41.968474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:41.968678 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:41.968791 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:41.969436 (+ 645us) server_negotiation.cc:299] Negotiation successful
0929 17:57:41.969595 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:57:41.970846 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:41.953042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:39613 (local address 127.7.118.2:44559)
0929 17:57:41.953659 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:41.953713 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:41.955301 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:41.956001 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:41.956010 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:41.956588 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:41.957174 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:41.957189 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:41.958166 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:41.958170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:41.959320 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:41.959332 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:41.959932 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:41.959937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:41.960140 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:41.960933 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:41.960953 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:41.965221 (+ 4268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:41.967412 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:41.967420 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:41.967442 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:41.967789 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:41.968123 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:41.968126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:41.968128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:41.968314 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:41.968806 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:41.968816 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:41.969132 (+ 316us) client_negotiation.cc:769] Sending connection context
0929 17:57:41.969349 (+ 217us) client_negotiation.cc:240] Negotiation successful
0929 17:57:41.969743 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":183,"threads_started":1}
I20260929 17:57:41.972329 8535 heartbeater.cc:347] Connected to a master server at 127.7.118.62:39613
I20260929 17:57:41.972652 8535 heartbeater.cc:464] Registering TS with master...
I20260929 17:57:41.973279 8535 heartbeater.cc:511] Master 127.7.118.62:39613 requested a full tablet report, sending...
I20260929 17:57:41.974888 8209 ts_manager.cc:194] Registered new tserver with Master: 51ce51bd45df46c59713ae4ae56fd0e0 (127.7.118.2:43109)
I20260929 17:57:41.975594 8209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.2@KRBTEST.COM'} at 127.7.118.2:44559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
I20260929 17:57:41.982990 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:0
--local_ip_for_outbound_sockets=127.7.118.3
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:39613
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:45203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:42.119494 8540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:42.119807 8540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:42.119865 8540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:42.124147 8540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:42.124241 8540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:42.124305 8540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 17:57:42.130237 8540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:45203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:39613
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:42.131385 8540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:42.132740 8540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:42.144899 8548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:42.144877 8545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:42.145107 8546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:42.146135 8540 server_base.cc:1034] running on GCE node
I20260929 17:57:42.146731 8540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:42.147918 8540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:42.149215 8540 hybrid_clock.cc:648] HybridClock initialized: now 1790704662149198 us; error 32 us; skew 500 ppm
Sep 29 17:57:42 dist-test-slave-r7ph krb5kdc[8165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704662, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:42.153689 8540 init.cc:377] Logged in from keytab as kudu/127.7.118.3@KRBTEST.COM (short username kudu)
I20260929 17:57:42.155510 8540 webserver.cc:460] Webserver started at http://127.7.118.3:40315/ using document root <none> and password file <none>
I20260929 17:57:42.156550 8540 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:42.156704 8540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:42.157048 8540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:42.159554 8540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-2/data/instance:
uuid: "a06ca375ebe74454ab3475583a026c0b"
format_stamp: "Formatted at 2026-09-29 17:57:42 on dist-test-slave-r7ph"
server_key: "e4af20de54312cea77278d1d3081cea8"
server_key_iv: "ef9dad43af346c40c3d75b6d214a600b"
server_key_version: "encryptionkey@0"
I20260929 17:57:42.160243 8540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance:
uuid: "a06ca375ebe74454ab3475583a026c0b"
format_stamp: "Formatted at 2026-09-29 17:57:42 on dist-test-slave-r7ph"
server_key: "e4af20de54312cea77278d1d3081cea8"
server_key_iv: "ef9dad43af346c40c3d75b6d214a600b"
server_key_version: "encryptionkey@0"
I20260929 17:57:42.164778 8540 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:57:42.168126 8555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:42.169813 8540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 17:57:42.170087 8540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "a06ca375ebe74454ab3475583a026c0b"
format_stamp: "Formatted at 2026-09-29 17:57:42 on dist-test-slave-r7ph"
server_key: "e4af20de54312cea77278d1d3081cea8"
server_key_iv: "ef9dad43af346c40c3d75b6d214a600b"
server_key_version: "encryptionkey@0"
I20260929 17:57:42.170243 8540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:42.195227 8540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:42.199406 8540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:42.199663 8540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:42.200330 8540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:57:42.201337 8540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:57:42.201413 8540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:42.201483 8540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:57:42.201527 8540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:42.212232 8540 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:42999
I20260929 17:57:42.212383 8668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:42999 every 8 connection(s)
I20260929 17:57:42.213357 8540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 17:57:42.221812 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 8540
I20260929 17:57:42.221979 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance
I20260929 17:57:42.222239 7640 external_mini_cluster.cc:1428] Setting key ce850af47e1b06c05d0da7371aabe482
Sep 29 17:57:42 dist-test-slave-r7ph krb5kdc[8165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704662, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:57:42.232012 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:42.216449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:60505 (local address 127.7.118.62:39613)
0929 17:57:42.216652 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:42.216658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:42.217769 (+ 1111us) server_negotiation.cc:407] Connection header received
0929 17:57:42.219511 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:42.219514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:42.219579 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:42.219666 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:42.221292 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.221874 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.223563 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.223836 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.223948 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:42.227606 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:42.227610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:42.227612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:42.227643 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:42.229485 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:42.230171 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:42.230175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:42.230176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:42.230329 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:42.230753 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:42.230757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:42.230759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:42.230975 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:42.231091 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:42.231687 (+ 596us) server_negotiation.cc:299] Negotiation successful
0929 17:57:42.231843 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260929 17:57:42.233390 8671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:42.216929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:39613 (local address 127.7.118.3:60505)
0929 17:57:42.217512 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:42.217573 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:42.219204 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:42.219931 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:42.219940 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:42.220367 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:42.220930 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:42.220945 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.222048 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.222053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:42.223416 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:42.223437 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.224084 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.224094 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:42.224488 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:42.225428 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:42.225447 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:42.227340 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:42.229664 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:42.229671 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:42.229687 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:42.230028 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:42.230438 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:42.230448 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:42.230450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:42.230637 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:42.231096 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:42.231106 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:42.231414 (+ 308us) client_negotiation.cc:769] Sending connection context
0929 17:57:42.231629 (+ 215us) client_negotiation.cc:240] Negotiation successful
0929 17:57:42.232038 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":184,"threads_started":1}
I20260929 17:57:42.234910 8669 heartbeater.cc:347] Connected to a master server at 127.7.118.62:39613
I20260929 17:57:42.235369 8669 heartbeater.cc:464] Registering TS with master...
I20260929 17:57:42.236068 8669 heartbeater.cc:511] Master 127.7.118.62:39613 requested a full tablet report, sending...
I20260929 17:57:42.237475 8209 ts_manager.cc:194] Registered new tserver with Master: a06ca375ebe74454ab3475583a026c0b (127.7.118.3:42999)
I20260929 17:57:42.238193 8209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.3@KRBTEST.COM'} at 127.7.118.3:60505
I20260929 17:57:42.238329 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:57:42.253847 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:42.243104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40676 (local address 127.7.118.62:39613)
0929 17:57:42.243274 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:42.243279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:42.243558 (+ 279us) server_negotiation.cc:407] Connection header received
0929 17:57:42.243615 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:42.243618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:42.243666 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:42.243751 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:42.244689 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.245239 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.246360 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.246598 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.246697 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:42.248972 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:42.248975 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:42.248977 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:42.249014 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:42.251338 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:42.252036 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:42.252040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:42.252041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:42.252170 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:42.252576 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:42.252579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:42.252581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:42.252844 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:42.252989 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:42.253510 (+ 521us) server_negotiation.cc:299] Negotiation successful
0929 17:57:42.253688 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 17:57:42.260011 8209 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:57:42.263293 8209 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:57:42.281250 8681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:42.274046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36322 (local address 127.7.118.1:45633)
0929 17:57:42.274612 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:42.274620 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:42.274881 (+ 261us) server_negotiation.cc:407] Connection header received
0929 17:57:42.275217 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:42.275226 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:42.275477 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:42.275614 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:42.276205 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.277446 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.279584 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.280249 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.280395 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:42.280574 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:42.280833 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 17:57:42.281076 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":103,"threads_started":1}
I20260929 17:57:42.281409 8685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:42.275319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57572 (local address 127.7.118.2:43109)
0929 17:57:42.276032 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:42.276043 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:42.276065 (+ 22us) server_negotiation.cc:407] Connection header received
0929 17:57:42.276139 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:42.276144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:42.276340 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:42.276475 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:42.276757 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.277847 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.279515 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.280586 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.280716 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:42.280880 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:42.281115 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 17:57:42.281238 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":141,"threads_started":1}
I20260929 17:57:42.281453 8683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:42.274895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:43109 (local address 127.0.0.1:57572)
0929 17:57:42.275708 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:42.275721 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:42.275800 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:42.276508 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:42.276513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:42.276535 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:42.276628 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:42.276634 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.277991 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.277994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:42.279379 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:42.279391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.280726 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.280730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:42.280848 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:42.280872 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 17:57:42.281078 (+ 206us) client_negotiation.cc:240] Negotiation successful
0929 17:57:42.281240 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":94,"thread_start_us":98,"threads_started":1}
I20260929 17:57:42.282027 8680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:42.273856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:45633 (local address 127.0.0.1:36322)
0929 17:57:42.274715 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:42.274777 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:42.275017 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:42.275845 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:42.275853 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:42.275876 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:42.276043 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:42.276050 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.277597 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.277601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:42.279379 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:42.279391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.280404 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.280407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:42.280543 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:42.280586 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 17:57:42.281805 (+ 1219us) client_negotiation.cc:240] Negotiation successful
0929 17:57:42.281891 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":113,"spinlock_wait_cycles":3328,"thread_start_us":192,"threads_started":1}
I20260929 17:57:42.282140 8684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:42.275013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49472 (local address 127.7.118.3:42999)
0929 17:57:42.275501 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:42.275517 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:42.275542 (+ 25us) server_negotiation.cc:407] Connection header received
0929 17:57:42.275622 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:42.275628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:42.275848 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:42.276024 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:42.276475 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.277553 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.279477 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.280586 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.280717 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:42.280880 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:42.281839 (+ 959us) server_negotiation.cc:299] Negotiation successful
0929 17:57:42.281982 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":154,"threads_started":1}
I20260929 17:57:42.281492 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:42.274067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:42999 (local address 127.0.0.1:49472)
0929 17:57:42.274848 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:42.274863 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:42.275039 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:42.276206 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:42.276213 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:42.276234 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:42.276339 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:42.276344 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.277719 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.277722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:42.279286 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:42.279297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.281050 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.281056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:42.281146 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:42.281171 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 17:57:42.281245 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 17:57:42.281319 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":143,"spinlock_wait_cycles":8960,"thread_start_us":102,"threads_started":1}
I20260929 17:57:42.284897 8469 tablet_service.cc:1467] Processing CreateTablet for tablet 1291d56513c94348b607aa6af9eaa021 (DEFAULT_TABLE table=test-table [id=53bfa7459fbc4d1e9057f650c7bea6a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:42.285283 8335 tablet_service.cc:1467] Processing CreateTablet for tablet 1291d56513c94348b607aa6af9eaa021 (DEFAULT_TABLE table=test-table [id=53bfa7459fbc4d1e9057f650c7bea6a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:42.286167 8469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1291d56513c94348b607aa6af9eaa021. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:42.286482 8335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1291d56513c94348b607aa6af9eaa021. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:42.288951 8603 tablet_service.cc:1467] Processing CreateTablet for tablet 1291d56513c94348b607aa6af9eaa021 (DEFAULT_TABLE table=test-table [id=53bfa7459fbc4d1e9057f650c7bea6a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:42.290068 8603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1291d56513c94348b607aa6af9eaa021. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:42.294965 8686 tablet_bootstrap.cc:492] T 1291d56513c94348b607aa6af9eaa021 P 51ce51bd45df46c59713ae4ae56fd0e0: Bootstrap starting.
I20260929 17:57:42.298786 8687 tablet_bootstrap.cc:492] T 1291d56513c94348b607aa6af9eaa021 P 58db145bc60f4dc5a4e373df3f138f5b: Bootstrap starting.
I20260929 17:57:42.299489 8688 tablet_bootstrap.cc:492] T 1291d56513c94348b607aa6af9eaa021 P a06ca375ebe74454ab3475583a026c0b: Bootstrap starting.
I20260929 17:57:42.299782 8686 tablet_bootstrap.cc:654] T 1291d56513c94348b607aa6af9eaa021 P 51ce51bd45df46c59713ae4ae56fd0e0: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:42.300896 8686 log.cc:824] T 1291d56513c94348b607aa6af9eaa021 P 51ce51bd45df46c59713ae4ae56fd0e0: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:42.301501 8687 tablet_bootstrap.cc:654] T 1291d56513c94348b607aa6af9eaa021 P 58db145bc60f4dc5a4e373df3f138f5b: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:42.301570 8688 tablet_bootstrap.cc:654] T 1291d56513c94348b607aa6af9eaa021 P a06ca375ebe74454ab3475583a026c0b: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:42.302485 8687 log.cc:824] T 1291d56513c94348b607aa6af9eaa021 P 58db145bc60f4dc5a4e373df3f138f5b: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:42.302816 8688 log.cc:824] T 1291d56513c94348b607aa6af9eaa021 P a06ca375ebe74454ab3475583a026c0b: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:42.303854 8686 tablet_bootstrap.cc:492] T 1291d56513c94348b607aa6af9eaa021 P 51ce51bd45df46c59713ae4ae56fd0e0: No bootstrap required, opened a new log
I20260929 17:57:42.304098 8686 ts_tablet_manager.cc:1397] T 1291d56513c94348b607aa6af9eaa021 P 51ce51bd45df46c59713ae4ae56fd0e0: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 17:57:42.304921 8687 tablet_bootstrap.cc:492] T 1291d56513c94348b607aa6af9eaa021 P 58db145bc60f4dc5a4e373df3f138f5b: No bootstrap required, opened a new log
I20260929 17:57:42.305147 8687 ts_tablet_manager.cc:1397] T 1291d56513c94348b607aa6af9eaa021 P 58db145bc60f4dc5a4e373df3f138f5b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 17:57:42.305199 8688 tablet_bootstrap.cc:492] T 1291d56513c94348b607aa6af9eaa021 P a06ca375ebe74454ab3475583a026c0b: No bootstrap required, opened a new log
I20260929 17:57:42.305377 8688 ts_tablet_manager.cc:1397] T 1291d56513c94348b607aa6af9eaa021 P a06ca375ebe74454ab3475583a026c0b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 17:57:42.308615 8686 raft_consensus.cc:348] T 1291d56513c94348b607aa6af9eaa021 P 51ce51bd45df46c59713ae4ae56fd0e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58db145bc60f4dc5a4e373df3f138f5b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45633 } } peers { permanent_uuid: "a06ca375ebe74454ab3475583a026c0b" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42999 } } peers { permanent_uuid: "51ce51bd45df46c59713ae4ae56fd0e0" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43109 } }
I20260929 17:57:42.309113 8686 raft_consensus.cc:374] T 1291d56513c94348b607aa6af9eaa021 P 51ce51bd45df46c59713ae4ae56fd0e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:42.309206 8686 raft_consensus.cc:729] T 1291d56513c94348b607aa6af9eaa021 P 51ce51bd45df46c59713ae4ae56fd0e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ce51bd45df46c59713ae4ae56fd0e0, State: Initialized, Role: FOLLOWER
I20260929 17:57:42.309370 8687 raft_consensus.cc:348] T 1291d56513c94348b607aa6af9eaa021 P 58db145bc60f4dc5a4e373df3f138f5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58db145bc60f4dc5a4e373df3f138f5b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45633 } } peers { permanent_uuid: "a06ca375ebe74454ab3475583a026c0b" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42999 } } peers { permanent_uuid: "51ce51bd45df46c59713ae4ae56fd0e0" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43109 } }
I20260929 17:57:42.309458 8688 raft_consensus.cc:348] T 1291d56513c94348b607aa6af9eaa021 P a06ca375ebe74454ab3475583a026c0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58db145bc60f4dc5a4e373df3f138f5b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45633 } } peers { permanent_uuid: "a06ca375ebe74454ab3475583a026c0b" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42999 } } peers { permanent_uuid: "51ce51bd45df46c59713ae4ae56fd0e0" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43109 } }
I20260929 17:57:42.309696 8687 raft_consensus.cc:374] T 1291d56513c94348b607aa6af9eaa021 P 58db145bc60f4dc5a4e373df3f138f5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:42.309747 8688 raft_consensus.cc:374] T 1291d56513c94348b607aa6af9eaa021 P a06ca375ebe74454ab3475583a026c0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:42.309751 8687 raft_consensus.cc:729] T 1291d56513c94348b607aa6af9eaa021 P 58db145bc60f4dc5a4e373df3f138f5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58db145bc60f4dc5a4e373df3f138f5b, State: Initialized, Role: FOLLOWER
I20260929 17:57:42.309803 8688 raft_consensus.cc:729] T 1291d56513c94348b607aa6af9eaa021 P a06ca375ebe74454ab3475583a026c0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a06ca375ebe74454ab3475583a026c0b, State: Initialized, Role: FOLLOWER
I20260929 17:57:42.309736 8686 consensus_queue.cc:260] T 1291d56513c94348b607aa6af9eaa021 P 51ce51bd45df46c59713ae4ae56fd0e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58db145bc60f4dc5a4e373df3f138f5b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45633 } } peers { permanent_uuid: "a06ca375ebe74454ab3475583a026c0b" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42999 } } peers { permanent_uuid: "51ce51bd45df46c59713ae4ae56fd0e0" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43109 } }
I20260929 17:57:42.310235 8687 consensus_queue.cc:260] T 1291d56513c94348b607aa6af9eaa021 P 58db145bc60f4dc5a4e373df3f138f5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58db145bc60f4dc5a4e373df3f138f5b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45633 } } peers { permanent_uuid: "a06ca375ebe74454ab3475583a026c0b" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42999 } } peers { permanent_uuid: "51ce51bd45df46c59713ae4ae56fd0e0" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43109 } }
I20260929 17:57:42.310328 8688 consensus_queue.cc:260] T 1291d56513c94348b607aa6af9eaa021 P a06ca375ebe74454ab3475583a026c0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58db145bc60f4dc5a4e373df3f138f5b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45633 } } peers { permanent_uuid: "a06ca375ebe74454ab3475583a026c0b" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42999 } } peers { permanent_uuid: "51ce51bd45df46c59713ae4ae56fd0e0" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43109 } }
I20260929 17:57:42.310806 8535 heartbeater.cc:503] Master 127.7.118.62:39613 was elected leader, sending a full tablet report...
I20260929 17:57:42.311088 8686 ts_tablet_manager.cc:1428] T 1291d56513c94348b607aa6af9eaa021 P 51ce51bd45df46c59713ae4ae56fd0e0: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 17:57:42.311357 8688 ts_tablet_manager.cc:1428] T 1291d56513c94348b607aa6af9eaa021 P a06ca375ebe74454ab3475583a026c0b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 17:57:42.311587 8669 heartbeater.cc:503] Master 127.7.118.62:39613 was elected leader, sending a full tablet report...
I20260929 17:57:42.312026 8401 heartbeater.cc:503] Master 127.7.118.62:39613 was elected leader, sending a full tablet report...
I20260929 17:57:42.312822 8687 ts_tablet_manager.cc:1428] T 1291d56513c94348b607aa6af9eaa021 P 58db145bc60f4dc5a4e373df3f138f5b: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260929 17:57:42.451248 8536 tablet.cc:2369] T 1291d56513c94348b607aa6af9eaa021 P 51ce51bd45df46c59713ae4ae56fd0e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:57:42.460131 8402 tablet.cc:2369] T 1291d56513c94348b607aa6af9eaa021 P 58db145bc60f4dc5a4e373df3f138f5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:57:42.464749 8670 tablet.cc:2369] T 1291d56513c94348b607aa6af9eaa021 P a06ca375ebe74454ab3475583a026c0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:57:42.498330 8694 raft_consensus.cc:482] T 1291d56513c94348b607aa6af9eaa021 P a06ca375ebe74454ab3475583a026c0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:57:42.498606 8694 raft_consensus.cc:504] T 1291d56513c94348b607aa6af9eaa021 P a06ca375ebe74454ab3475583a026c0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58db145bc60f4dc5a4e373df3f138f5b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45633 } } peers { permanent_uuid: "a06ca375ebe74454ab3475583a026c0b" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42999 } } peers { permanent_uuid: "51ce51bd45df46c59713ae4ae56fd0e0" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43109 } }
I20260929 17:57:42.499833 8694 leader_election.cc:288] T 1291d56513c94348b607aa6af9eaa021 P a06ca375ebe74454ab3475583a026c0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58db145bc60f4dc5a4e373df3f138f5b (127.7.118.1:45633), 51ce51bd45df46c59713ae4ae56fd0e0 (127.7.118.2:43109)
I20260929 17:57:42.504360 8681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:42.500351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:53379 (local address 127.7.118.1:45633)
0929 17:57:42.500497 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:42.500502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:42.500712 (+ 210us) server_negotiation.cc:407] Connection header received
0929 17:57:42.500843 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:42.500846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:42.500902 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:42.500985 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:42.501310 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.501860 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.503138 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.503644 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.503781 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:42.503845 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:42.504141 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 17:57:42.504225 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 17:57:42.504391 8685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:42.500352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:36017 (local address 127.7.118.2:43109)
0929 17:57:42.500539 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:42.500544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:42.500924 (+ 380us) server_negotiation.cc:407] Connection header received
0929 17:57:42.501008 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:42.501011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:42.501062 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:42.501147 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:42.501447 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.502021 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.503206 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.503619 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.503776 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:42.503864 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:42.504141 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 17:57:42.504225 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260929 17:57:42.504388 8695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:42.500217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:43109 (local address 127.7.118.3:36017)
0929 17:57:42.500740 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:42.500755 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:42.500883 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:42.501172 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:42.501174 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:42.501188 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:42.501312 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:42.501319 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.502166 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.502169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:42.503032 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:42.503086 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.503798 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.503802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:42.503960 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:42.503989 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 17:57:42.504078 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 17:57:42.504198 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":98,"thread_start_us":103,"threads_started":1}
I20260929 17:57:42.504388 8671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:42.500217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:45633 (local address 127.7.118.3:53379)
0929 17:57:42.500451 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:42.500469 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:42.500729 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:42.501014 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:42.501017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:42.501035 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:42.501176 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:42.501183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.501978 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.501982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:42.502980 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:42.502990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.503898 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.503901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:42.503997 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:42.504014 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 17:57:42.504087 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 17:57:42.504198 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":8}
I20260929 17:57:42.505153 8489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1291d56513c94348b607aa6af9eaa021" candidate_uuid: "a06ca375ebe74454ab3475583a026c0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ce51bd45df46c59713ae4ae56fd0e0" is_pre_election: true
I20260929 17:57:42.505324 8355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1291d56513c94348b607aa6af9eaa021" candidate_uuid: "a06ca375ebe74454ab3475583a026c0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58db145bc60f4dc5a4e373df3f138f5b" is_pre_election: true
I20260929 17:57:42.505524 8489 raft_consensus.cc:2454] T 1291d56513c94348b607aa6af9eaa021 P 51ce51bd45df46c59713ae4ae56fd0e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a06ca375ebe74454ab3475583a026c0b in term 0.
I20260929 17:57:42.505657 8355 raft_consensus.cc:2454] T 1291d56513c94348b607aa6af9eaa021 P 58db145bc60f4dc5a4e373df3f138f5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a06ca375ebe74454ab3475583a026c0b in term 0.
I20260929 17:57:42.506080 8556 leader_election.cc:302] T 1291d56513c94348b607aa6af9eaa021 P a06ca375ebe74454ab3475583a026c0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51ce51bd45df46c59713ae4ae56fd0e0, a06ca375ebe74454ab3475583a026c0b; no voters:
I20260929 17:57:42.506423 8694 raft_consensus.cc:2781] T 1291d56513c94348b607aa6af9eaa021 P a06ca375ebe74454ab3475583a026c0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:57:42.506536 8694 raft_consensus.cc:482] T 1291d56513c94348b607aa6af9eaa021 P a06ca375ebe74454ab3475583a026c0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:57:42.506595 8694 raft_consensus.cc:3037] T 1291d56513c94348b607aa6af9eaa021 P a06ca375ebe74454ab3475583a026c0b [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:42.507676 8694 raft_consensus.cc:504] T 1291d56513c94348b607aa6af9eaa021 P a06ca375ebe74454ab3475583a026c0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58db145bc60f4dc5a4e373df3f138f5b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45633 } } peers { permanent_uuid: "a06ca375ebe74454ab3475583a026c0b" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42999 } } peers { permanent_uuid: "51ce51bd45df46c59713ae4ae56fd0e0" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43109 } }
I20260929 17:57:42.508152 8694 leader_election.cc:288] T 1291d56513c94348b607aa6af9eaa021 P a06ca375ebe74454ab3475583a026c0b [CANDIDATE]: Term 1 election: Requested vote from peers 58db145bc60f4dc5a4e373df3f138f5b (127.7.118.1:45633), 51ce51bd45df46c59713ae4ae56fd0e0 (127.7.118.2:43109)
I20260929 17:57:42.508579 8355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1291d56513c94348b607aa6af9eaa021" candidate_uuid: "a06ca375ebe74454ab3475583a026c0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58db145bc60f4dc5a4e373df3f138f5b"
I20260929 17:57:42.508715 8355 raft_consensus.cc:3037] T 1291d56513c94348b607aa6af9eaa021 P 58db145bc60f4dc5a4e373df3f138f5b [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:42.508723 8489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1291d56513c94348b607aa6af9eaa021" candidate_uuid: "a06ca375ebe74454ab3475583a026c0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ce51bd45df46c59713ae4ae56fd0e0"
I20260929 17:57:42.508896 8489 raft_consensus.cc:3037] T 1291d56513c94348b607aa6af9eaa021 P 51ce51bd45df46c59713ae4ae56fd0e0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:42.509924 8355 raft_consensus.cc:2454] T 1291d56513c94348b607aa6af9eaa021 P 58db145bc60f4dc5a4e373df3f138f5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a06ca375ebe74454ab3475583a026c0b in term 1.
I20260929 17:57:42.510061 8489 raft_consensus.cc:2454] T 1291d56513c94348b607aa6af9eaa021 P 51ce51bd45df46c59713ae4ae56fd0e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a06ca375ebe74454ab3475583a026c0b in term 1.
I20260929 17:57:42.510371 8559 leader_election.cc:302] T 1291d56513c94348b607aa6af9eaa021 P a06ca375ebe74454ab3475583a026c0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58db145bc60f4dc5a4e373df3f138f5b, a06ca375ebe74454ab3475583a026c0b; no voters:
I20260929 17:57:42.510692 8694 raft_consensus.cc:2781] T 1291d56513c94348b607aa6af9eaa021 P a06ca375ebe74454ab3475583a026c0b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:42.510946 8694 raft_consensus.cc:686] T 1291d56513c94348b607aa6af9eaa021 P a06ca375ebe74454ab3475583a026c0b [term 1 LEADER]: Becoming Leader. State: Replica: a06ca375ebe74454ab3475583a026c0b, State: Running, Role: LEADER
I20260929 17:57:42.511427 8694 consensus_queue.cc:237] T 1291d56513c94348b607aa6af9eaa021 P a06ca375ebe74454ab3475583a026c0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58db145bc60f4dc5a4e373df3f138f5b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45633 } } peers { permanent_uuid: "a06ca375ebe74454ab3475583a026c0b" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42999 } } peers { permanent_uuid: "51ce51bd45df46c59713ae4ae56fd0e0" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43109 } }
I20260929 17:57:42.515550 8208 catalog_manager.cc:5495] T 1291d56513c94348b607aa6af9eaa021 P a06ca375ebe74454ab3475583a026c0b reported cstate change: term changed from 0 to 1, leader changed from <none> to a06ca375ebe74454ab3475583a026c0b (127.7.118.3). New cstate: current_term: 1 leader_uuid: "a06ca375ebe74454ab3475583a026c0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58db145bc60f4dc5a4e373df3f138f5b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a06ca375ebe74454ab3475583a026c0b" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51ce51bd45df46c59713ae4ae56fd0e0" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43109 } health_report { overall_health: UNKNOWN } } }
Sep 29 17:57:42 dist-test-slave-r7ph krb5kdc[8165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.1@KRBTEST.COM
I20260929 17:57:42.583570 8681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:42.572650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36338 (local address 127.7.118.1:45633)
0929 17:57:42.572933 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:42.572940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:42.573044 (+ 104us) server_negotiation.cc:407] Connection header received
0929 17:57:42.573242 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:42.573246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:42.573301 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:42.573396 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:42.574284 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.574864 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.575866 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.576113 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.576237 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:42.579147 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:42.579162 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:42.579177 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:42.579214 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:42.581301 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:42.581898 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:42.581905 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:42.581907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:42.582009 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:42.582475 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:42.582479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:42.582480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:42.582766 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:42.582963 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:42.583297 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 17:57:42.583437 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
Sep 29 17:57:42 dist-test-slave-r7ph krb5kdc[8165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704662, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 17:57:42 dist-test-slave-r7ph krb5kdc[8165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704662, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.118.1@KRBTEST.COM
I20260929 17:57:42.614416 8681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:42.604110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36342 (local address 127.7.118.1:45633)
0929 17:57:42.604344 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:42.604350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:42.604549 (+ 199us) server_negotiation.cc:407] Connection header received
0929 17:57:42.604803 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:42.604807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:42.604853 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:42.604947 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:42.605704 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.606311 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.607201 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.607454 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.607564 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:42.610612 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:42.610618 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:42.610623 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:42.610666 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:42.612415 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:42.612993 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:42.612996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:42.612997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:42.613082 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:42.613345 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:42.613348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:42.613349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:42.613538 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:42.613646 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:42.614010 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 17:57:42.614179 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
Sep 29 17:57:42 dist-test-slave-r7ph krb5kdc[8165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 17:57:42 dist-test-slave-r7ph krb5kdc[8165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.1@KRBTEST.COM
I20260929 17:57:42.648242 8681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:42.635873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36348 (local address 127.7.118.1:45633)
0929 17:57:42.636171 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:42.636177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:42.636390 (+ 213us) server_negotiation.cc:407] Connection header received
0929 17:57:42.636456 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:42.636459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:42.636510 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:42.636580 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:42.637782 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.638591 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.639766 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:42.640016 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:42.640234 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:42.643350 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:42.643355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:42.643358 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:42.643400 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:42.645712 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:42.646512 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:42.646515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:42.646517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:42.646648 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:42.647102 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:42.647105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:42.647106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:42.647305 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:42.647436 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:42.647806 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 17:57:42.648022 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260929 17:57:42.649873 8315 tablet_service.cc:2908] Rejecting scan request for tablet 1291d56513c94348b607aa6af9eaa021: Uninitialized: clean time has not yet been initialized
I20260929 17:57:42.652549 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 8272
I20260929 17:57:42.658680 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 8406
I20260929 17:57:42.665072 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 8540
I20260929 17:57:42.671463 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 8178
2026-09-29T17:57:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2687 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:42 dist-test-slave-r7ph krb5kdc[8710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:42 dist-test-slave-r7ph krb5kdc[8710](info): set up 2 sockets
Sep 29 17:57:42 dist-test-slave-r7ph krb5kdc[8710](info): commencing operation
krb5kdc: starting...
W20260929 17:57:43.758212 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 17:57:43 dist-test-slave-r7ph krb5kdc[8710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 17:57:43.911218 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:33793
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:33793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:44.033437 8723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:44.033852 8723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:44.033946 8723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:44.038534 8723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:44.038641 8723 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 17:57:44.038673 8723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:44.038717 8723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:44.038784 8723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:44.038843 8723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:44.045267 8723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:33793
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:33793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:44.046648 8723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:44.048220 8723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:44.058027 8728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:44.058179 8723 server_base.cc:1034] running on GCE node
W20260929 17:57:44.058094 8729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:44.058018 8731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:44.059103 8723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:44.060537 8723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:44.061791 8723 hybrid_clock.cc:648] HybridClock initialized: now 1790704664061758 us; error 63 us; skew 500 ppm
Sep 29 17:57:44 dist-test-slave-r7ph krb5kdc[8710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704664, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:44.065675 8723 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 17:57:44.066854 8723 webserver.cc:460] Webserver started at http://127.7.118.62:39501/ using document root <none> and password file <none>
I20260929 17:57:44.067488 8723 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:44.067564 8723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:44.067761 8723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:44.070386 8723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "acb652fa8bad4fc99cc18238f6297523"
format_stamp: "Formatted at 2026-09-29 17:57:44 on dist-test-slave-r7ph"
server_key: "e829c7a50caf55df21a5430546db5e9f"
server_key_iv: "c047e63788992645232e008ac1335882"
server_key_version: "encryptionkey@0"
I20260929 17:57:44.071017 8723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "acb652fa8bad4fc99cc18238f6297523"
format_stamp: "Formatted at 2026-09-29 17:57:44 on dist-test-slave-r7ph"
server_key: "e829c7a50caf55df21a5430546db5e9f"
server_key_iv: "c047e63788992645232e008ac1335882"
server_key_version: "encryptionkey@0"
I20260929 17:57:44.075248 8723 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 17:57:44.078076 8738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:44.079679 8723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 17:57:44.079857 8723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "acb652fa8bad4fc99cc18238f6297523"
format_stamp: "Formatted at 2026-09-29 17:57:44 on dist-test-slave-r7ph"
server_key: "e829c7a50caf55df21a5430546db5e9f"
server_key_iv: "c047e63788992645232e008ac1335882"
server_key_version: "encryptionkey@0"
I20260929 17:57:44.080019 8723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:44.099323 8723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:44.103191 8723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:44.103415 8723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:44.112254 8723 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:33793
I20260929 17:57:44.112293 8800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:33793 every 8 connection(s)
I20260929 17:57:44.113379 8723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:44.116642 8801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:44.119483 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 8723
I20260929 17:57:44.119639 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 17:57:44.119948 7640 external_mini_cluster.cc:1428] Setting key c203ed8f26857ff50b8f692f6cf174b5
I20260929 17:57:44.124529 8801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acb652fa8bad4fc99cc18238f6297523: Bootstrap starting.
I20260929 17:57:44.127388 8801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acb652fa8bad4fc99cc18238f6297523: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:44.128320 8801 log.cc:824] T 00000000000000000000000000000000 P acb652fa8bad4fc99cc18238f6297523: Log is configured to *not* fsync() on all Append() calls
Sep 29 17:57:44 dist-test-slave-r7ph krb5kdc[8710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:57:44.131222 8801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acb652fa8bad4fc99cc18238f6297523: No bootstrap required, opened a new log
I20260929 17:57:44.136687 8801 raft_consensus.cc:348] T 00000000000000000000000000000000 P acb652fa8bad4fc99cc18238f6297523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb652fa8bad4fc99cc18238f6297523" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33793 } }
I20260929 17:57:44.137058 8801 raft_consensus.cc:374] T 00000000000000000000000000000000 P acb652fa8bad4fc99cc18238f6297523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:44.137156 8801 raft_consensus.cc:729] T 00000000000000000000000000000000 P acb652fa8bad4fc99cc18238f6297523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb652fa8bad4fc99cc18238f6297523, State: Initialized, Role: FOLLOWER
I20260929 17:57:44.137728 8801 consensus_queue.cc:260] T 00000000000000000000000000000000 P acb652fa8bad4fc99cc18238f6297523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb652fa8bad4fc99cc18238f6297523" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33793 } }
I20260929 17:57:44.137934 8801 raft_consensus.cc:388] T 00000000000000000000000000000000 P acb652fa8bad4fc99cc18238f6297523 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:44.138000 8801 raft_consensus.cc:482] T 00000000000000000000000000000000 P acb652fa8bad4fc99cc18238f6297523 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:44.138103 8801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acb652fa8bad4fc99cc18238f6297523 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:44.139016 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:44.121791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57420 (local address 127.7.118.62:33793)
0929 17:57:44.122446 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:44.122463 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:44.122505 (+ 42us) server_negotiation.cc:407] Connection header received
0929 17:57:44.123336 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:44.123359 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:44.123746 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:44.124346 (+ 600us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:44.125326 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:44.126481 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:44.127425 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:44.128045 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:44.128248 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:44.131499 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:44.131506 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:44.131524 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:44.131559 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:44.134143 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:44.134771 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:44.134778 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:44.134784 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:44.134964 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:44.136015 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:44.136023 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:44.136025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:44.136565 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:44.137002 (+ 437us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:44.137360 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 17:57:44.137734 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":161,"threads_started":1}
I20260929 17:57:44.139461 8801 raft_consensus.cc:504] T 00000000000000000000000000000000 P acb652fa8bad4fc99cc18238f6297523 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb652fa8bad4fc99cc18238f6297523" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33793 } }
I20260929 17:57:44.140009 8801 leader_election.cc:302] T 00000000000000000000000000000000 P acb652fa8bad4fc99cc18238f6297523 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acb652fa8bad4fc99cc18238f6297523; no voters:
I20260929 17:57:44.140409 8801 leader_election.cc:288] T 00000000000000000000000000000000 P acb652fa8bad4fc99cc18238f6297523 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:44.140825 8807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acb652fa8bad4fc99cc18238f6297523 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:44.141181 8807 raft_consensus.cc:686] T 00000000000000000000000000000000 P acb652fa8bad4fc99cc18238f6297523 [term 1 LEADER]: Becoming Leader. State: Replica: acb652fa8bad4fc99cc18238f6297523, State: Running, Role: LEADER
I20260929 17:57:44.141572 8807 consensus_queue.cc:237] T 00000000000000000000000000000000 P acb652fa8bad4fc99cc18238f6297523 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb652fa8bad4fc99cc18238f6297523" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33793 } }
I20260929 17:57:44.142247 8801 sys_catalog.cc:564] T 00000000000000000000000000000000 P acb652fa8bad4fc99cc18238f6297523 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:44.143534 8806 sys_catalog.cc:455] T 00000000000000000000000000000000 P acb652fa8bad4fc99cc18238f6297523 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acb652fa8bad4fc99cc18238f6297523" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb652fa8bad4fc99cc18238f6297523" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33793 } } }
I20260929 17:57:44.143699 8806 sys_catalog.cc:458] T 00000000000000000000000000000000 P acb652fa8bad4fc99cc18238f6297523 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:44.144002 8808 sys_catalog.cc:455] T 00000000000000000000000000000000 P acb652fa8bad4fc99cc18238f6297523 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acb652fa8bad4fc99cc18238f6297523. Latest consensus state: current_term: 1 leader_uuid: "acb652fa8bad4fc99cc18238f6297523" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb652fa8bad4fc99cc18238f6297523" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33793 } } }
I20260929 17:57:44.144117 8808 sys_catalog.cc:458] T 00000000000000000000000000000000 P acb652fa8bad4fc99cc18238f6297523 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:44.144542 8815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:44.148214 8815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:44.155251 8815 catalog_manager.cc:1347] Generated new cluster ID: ddfa755432f04e679e465e0b3533a0cb
I20260929 17:57:44.155361 8815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:44.173960 8815 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:44.175431 8815 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:44.185961 8815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acb652fa8bad4fc99cc18238f6297523: Generated new TSK 0
I20260929 17:57:44.186910 8815 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.118.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
I20260929 17:57:44.235004 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:0
--local_ip_for_outbound_sockets=127.7.118.1
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:33793
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:44.359445 8827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:44.359788 8827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:44.359841 8827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:44.365010 8827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:44.365106 8827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:44.365154 8827 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 17:57:44.365227 8827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 17:57:44.371855 8827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.118.62:33793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:44.373139 8827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:44.374437 8827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:44.385730 8833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:44.385730 8835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:44.385812 8832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:44.386309 8827 server_base.cc:1034] running on GCE node
I20260929 17:57:44.386747 8827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:44.387424 8827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:44.388672 8827 hybrid_clock.cc:648] HybridClock initialized: now 1790704664388658 us; error 33 us; skew 500 ppm
Sep 29 17:57:44 dist-test-slave-r7ph krb5kdc[8710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704664, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:44.392069 8827 init.cc:377] Logged in from keytab as kudu/127.7.118.1@KRBTEST.COM (short username kudu)
I20260929 17:57:44.393138 8827 webserver.cc:460] Webserver started at http://127.7.118.1:44951/ using document root <none> and password file <none>
I20260929 17:57:44.393760 8827 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:44.393846 8827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:44.394079 8827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:44.396445 8827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-0/data/instance:
uuid: "8e9d2970919a488da31c348bc8bf2d8e"
format_stamp: "Formatted at 2026-09-29 17:57:44 on dist-test-slave-r7ph"
server_key: "32ca2e69fde79d1d9204e5e35ea4fc08"
server_key_iv: "8a176dcbffc2537c6a4787a4c7ac89f8"
server_key_version: "encryptionkey@0"
I20260929 17:57:44.397114 8827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance:
uuid: "8e9d2970919a488da31c348bc8bf2d8e"
format_stamp: "Formatted at 2026-09-29 17:57:44 on dist-test-slave-r7ph"
server_key: "32ca2e69fde79d1d9204e5e35ea4fc08"
server_key_iv: "8a176dcbffc2537c6a4787a4c7ac89f8"
server_key_version: "encryptionkey@0"
I20260929 17:57:44.401680 8827 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260929 17:57:44.405531 8842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:44.407260 8827 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 17:57:44.407431 8827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "8e9d2970919a488da31c348bc8bf2d8e"
format_stamp: "Formatted at 2026-09-29 17:57:44 on dist-test-slave-r7ph"
server_key: "32ca2e69fde79d1d9204e5e35ea4fc08"
server_key_iv: "8a176dcbffc2537c6a4787a4c7ac89f8"
server_key_version: "encryptionkey@0"
I20260929 17:57:44.407588 8827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:44.433913 8827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:44.437883 8827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:44.438143 8827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:44.439754 8827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:57:44.439909 8827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:44.440008 8827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:57:44.440060 8827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 17:57:44 dist-test-slave-r7ph krb5kdc[8710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704664, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:57:44.451710 8827 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:42559
I20260929 17:57:44.451861 8957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:42559 every 8 connection(s)
I20260929 17:57:44.452718 8827 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 17:57:44.453663 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 8827
I20260929 17:57:44.453933 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance
I20260929 17:57:44.454427 7640 external_mini_cluster.cc:1428] Setting key 18e00443d7cdb737b82ecfc9748ed622
I20260929 17:57:44.457072 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:44.440883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:57129 (local address 127.7.118.62:33793)
0929 17:57:44.441012 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:44.441016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:44.442491 (+ 1475us) server_negotiation.cc:407] Connection header received
0929 17:57:44.444047 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:44.444051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:44.444129 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:44.444245 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:44.445999 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:44.446987 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:44.448443 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:44.448699 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:44.448824 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:44.452311 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:44.452315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:44.452317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:44.452347 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:44.454375 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:44.455121 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:44.455125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:44.455127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:44.455231 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:44.455669 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:44.455671 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:44.455672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:44.455918 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:44.456038 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:44.456604 (+ 566us) server_negotiation.cc:299] Negotiation successful
0929 17:57:44.456768 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 17:57:44.457731 8851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:44.441577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33793 (local address 127.7.118.1:57129)
0929 17:57:44.442322 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:44.442378 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:44.443678 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:44.444429 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:44.444437 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:44.444983 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:44.445776 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:44.445801 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:44.447184 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:44.447189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:44.448293 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:44.448304 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:44.448894 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:44.448899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:44.449103 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:44.449808 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:44.449835 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:44.452054 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:44.454570 (+ 2516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:44.454577 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:44.454593 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:44.455002 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:44.455378 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:44.455381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:44.455383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:44.455573 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:44.456042 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:44.456051 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:44.456342 (+ 291us) client_negotiation.cc:769] Sending connection context
0929 17:57:44.456514 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 17:57:44.456805 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":284,"threads_started":1}
I20260929 17:57:44.459125 8958 heartbeater.cc:347] Connected to a master server at 127.7.118.62:33793
I20260929 17:57:44.459318 8958 heartbeater.cc:464] Registering TS with master...
I20260929 17:57:44.459875 8958 heartbeater.cc:511] Master 127.7.118.62:33793 requested a full tablet report, sending...
I20260929 17:57:44.461634 8754 ts_manager.cc:194] Registered new tserver with Master: 8e9d2970919a488da31c348bc8bf2d8e (127.7.118.1:42559)
I20260929 17:57:44.462961 8754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.1@KRBTEST.COM'} at 127.7.118.1:57129
WARNING: no policy specified for kudu/127.7.118.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.2@KRBTEST.COM" created.
I20260929 17:57:44.472538 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:44.461960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33793 (local address 127.7.118.1:43331)
0929 17:57:44.463802 (+ 1842us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:44.463820 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:44.463933 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:44.464214 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:44.464217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:44.464443 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:44.464545 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:44.464550 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:44.465331 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:44.465334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:44.466596 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:44.466608 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:44.467116 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:44.467119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:44.467216 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:44.468119 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:44.468132 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:44.468587 (+ 455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:44.470850 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:44.470856 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:44.470858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:44.471088 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:44.471410 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:44.471413 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:44.471415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:44.471510 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:44.471922 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:44.471925 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:44.472036 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 17:57:44.472146 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 17:57:44.472339 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1743,"mutex_wait_us":3,"thread_start_us":114,"threads_started":1}
I20260929 17:57:44.472525 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:44.462080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:43331 (local address 127.7.118.62:33793)
0929 17:57:44.462229 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:44.462236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:44.463912 (+ 1676us) server_negotiation.cc:407] Connection header received
0929 17:57:44.464032 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:44.464035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:44.464081 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:44.464192 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:44.464663 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:44.465182 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:44.466735 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:44.466985 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:44.467106 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:44.468815 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:44.468822 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:44.468829 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:44.468866 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:44.470635 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:44.471204 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:44.471208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:44.471210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:44.471302 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:44.471600 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:44.471605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:44.471607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:44.471820 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:44.471934 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:44.472217 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 17:57:44.472338 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
I20260929 17:57:44.487273 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:0
--local_ip_for_outbound_sockets=127.7.118.2
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:33793
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:44.609367 8967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:44.609681 8967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:44.609735 8967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:44.614198 8967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:44.614297 8967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:44.614341 8967 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 17:57:44.614377 8967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 17:57:44.620033 8967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.118.62:33793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:44.621209 8967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:44.622277 8967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:44.633167 8975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:44.633145 8973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:44.633145 8972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:44.633682 8967 server_base.cc:1034] running on GCE node
I20260929 17:57:44.634598 8967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:44.635394 8967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:44.636581 8967 hybrid_clock.cc:648] HybridClock initialized: now 1790704664636538 us; error 61 us; skew 500 ppm
Sep 29 17:57:44 dist-test-slave-r7ph krb5kdc[8710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704664, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:44.640168 8967 init.cc:377] Logged in from keytab as kudu/127.7.118.2@KRBTEST.COM (short username kudu)
I20260929 17:57:44.641247 8967 webserver.cc:460] Webserver started at http://127.7.118.2:45247/ using document root <none> and password file <none>
I20260929 17:57:44.641898 8967 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:44.641973 8967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:44.642204 8967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:44.645117 8967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-1/data/instance:
uuid: "2c2215bc64b14d40b64835326a0cebbd"
format_stamp: "Formatted at 2026-09-29 17:57:44 on dist-test-slave-r7ph"
server_key: "f32a1f7a2110cd77f438fc55f3013b14"
server_key_iv: "fe7111cf44521dc3f49ff3330876c623"
server_key_version: "encryptionkey@0"
I20260929 17:57:44.645759 8967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance:
uuid: "2c2215bc64b14d40b64835326a0cebbd"
format_stamp: "Formatted at 2026-09-29 17:57:44 on dist-test-slave-r7ph"
server_key: "f32a1f7a2110cd77f438fc55f3013b14"
server_key_iv: "fe7111cf44521dc3f49ff3330876c623"
server_key_version: "encryptionkey@0"
I20260929 17:57:44.650048 8967 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 17:57:44.653128 8982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:44.655046 8967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:57:44.655228 8967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "2c2215bc64b14d40b64835326a0cebbd"
format_stamp: "Formatted at 2026-09-29 17:57:44 on dist-test-slave-r7ph"
server_key: "f32a1f7a2110cd77f438fc55f3013b14"
server_key_iv: "fe7111cf44521dc3f49ff3330876c623"
server_key_version: "encryptionkey@0"
I20260929 17:57:44.655417 8967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:44.669905 8967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:44.674208 8967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:44.674516 8967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:44.676090 8967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:57:44.676146 8967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:44.676188 8967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:57:44.676244 8967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:44.689039 8967 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:43843
Sep 29 17:57:44 dist-test-slave-r7ph krb5kdc[8710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704664, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:57:44.689172 9097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:43843 every 8 connection(s)
I20260929 17:57:44.690111 8967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 17:57:44.694100 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:44.677157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:37907 (local address 127.7.118.62:33793)
0929 17:57:44.677315 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:44.677321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:44.678312 (+ 991us) server_negotiation.cc:407] Connection header received
0929 17:57:44.679685 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:44.679689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:44.679754 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:44.679888 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:44.681963 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:44.682961 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:44.685536 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:44.685850 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:44.685969 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:44.689960 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:44.689964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:44.689967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:44.690006 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:44.691795 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:44.692417 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:44.692422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:44.692424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:44.692519 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:44.692900 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:44.692904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:44.692906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:44.693123 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:44.693232 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:44.693779 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 17:57:44.693928 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 17:57:44.695128 8991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:44.677632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33793 (local address 127.7.118.2:37907)
0929 17:57:44.678110 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:44.678166 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:44.679410 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:44.680152 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:44.680162 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:44.680785 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:44.681709 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:44.681740 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:44.683136 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:44.683140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:44.685272 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:44.685289 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:44.686091 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:44.686100 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:44.686464 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:44.687515 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:44.687545 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:44.689720 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:44.691950 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:44.691957 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:44.691977 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:44.692295 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:44.692624 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:44.692627 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:44.692629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:44.692801 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:44.693233 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:44.693241 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:44.693530 (+ 289us) client_negotiation.cc:769] Sending connection context
0929 17:57:44.693737 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 17:57:44.694131 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":152,"threads_started":1}
I20260929 17:57:44.695158 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 8967
I20260929 17:57:44.695281 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance
I20260929 17:57:44.695608 7640 external_mini_cluster.cc:1428] Setting key d90035500b3ae75dde12d67fd92b113e
I20260929 17:57:44.696909 9098 heartbeater.cc:347] Connected to a master server at 127.7.118.62:33793
I20260929 17:57:44.697100 9098 heartbeater.cc:464] Registering TS with master...
I20260929 17:57:44.697662 9098 heartbeater.cc:511] Master 127.7.118.62:33793 requested a full tablet report, sending...
I20260929 17:57:44.699052 8755 ts_manager.cc:194] Registered new tserver with Master: 2c2215bc64b14d40b64835326a0cebbd (127.7.118.2:43843)
I20260929 17:57:44.699998 8755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.2@KRBTEST.COM'} at 127.7.118.2:37907
WARNING: no policy specified for kudu/127.7.118.3@KRBTEST.COM; defaulting to no policy
I20260929 17:57:44.708449 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:44.700058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:47013 (local address 127.7.118.62:33793)
0929 17:57:44.700186 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:44.700191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:44.700209 (+ 18us) server_negotiation.cc:407] Connection header received
0929 17:57:44.700418 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:44.700421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:44.700468 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:44.700519 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:44.701217 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:44.701875 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:44.702912 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:44.703174 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:44.703299 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:44.704774 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:44.704778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:44.704781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:44.704820 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:44.706566 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:44.707059 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:44.707063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:44.707065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:44.707161 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:44.707456 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:44.707461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:44.707462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:44.707704 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:44.707817 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:44.708150 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 17:57:44.708265 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 17:57:44.708463 9105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:44.699711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33793 (local address 127.7.118.2:47013)
0929 17:57:44.700068 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:44.700112 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:44.700224 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:44.700639 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:44.700641 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:44.700909 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:44.701067 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:44.701075 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:44.702034 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:44.702038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:44.702757 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:44.702768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:44.703319 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:44.703323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:44.703446 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:44.704163 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:44.704186 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:44.704595 (+ 409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:44.706694 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:44.706701 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:44.706705 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:44.706942 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:44.707259 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:44.707263 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:44.707265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:44.707362 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:44.707805 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:44.707808 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:44.707963 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 17:57:44.708081 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 17:57:44.708261 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":8,"thread_start_us":152,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
I20260929 17:57:44.731305 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:0
--local_ip_for_outbound_sockets=127.7.118.3
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:33793
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:44.869270 9107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:44.869647 9107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:44.869735 9107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:44.875509 9107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:44.875658 9107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:44.875746 9107 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 17:57:44.875839 9107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 17:57:44.881855 9107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.118.62:33793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:44.883198 9107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:44.884377 9107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:44.895536 9115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:44.895565 9113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:44.895545 9112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:44.896111 9107 server_base.cc:1034] running on GCE node
I20260929 17:57:44.896667 9107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:44.897444 9107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:44.898646 9107 hybrid_clock.cc:648] HybridClock initialized: now 1790704664898598 us; error 80 us; skew 500 ppm
Sep 29 17:57:44 dist-test-slave-r7ph krb5kdc[8710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704664, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:44.902100 9107 init.cc:377] Logged in from keytab as kudu/127.7.118.3@KRBTEST.COM (short username kudu)
I20260929 17:57:44.903247 9107 webserver.cc:460] Webserver started at http://127.7.118.3:34161/ using document root <none> and password file <none>
I20260929 17:57:44.903945 9107 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:44.904024 9107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:44.904278 9107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:44.907061 9107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-2/data/instance:
uuid: "7df4a7cb612e48848f68934bcc690309"
format_stamp: "Formatted at 2026-09-29 17:57:44 on dist-test-slave-r7ph"
server_key: "e59cbe715cbf2c9f5b47e841e5ea8498"
server_key_iv: "bfcb6f15f44a2b07d77fa129c3330747"
server_key_version: "encryptionkey@0"
I20260929 17:57:44.907706 9107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance:
uuid: "7df4a7cb612e48848f68934bcc690309"
format_stamp: "Formatted at 2026-09-29 17:57:44 on dist-test-slave-r7ph"
server_key: "e59cbe715cbf2c9f5b47e841e5ea8498"
server_key_iv: "bfcb6f15f44a2b07d77fa129c3330747"
server_key_version: "encryptionkey@0"
I20260929 17:57:44.912895 9107 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 17:57:44.916040 9122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:44.917568 9107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 17:57:44.917758 9107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "7df4a7cb612e48848f68934bcc690309"
format_stamp: "Formatted at 2026-09-29 17:57:44 on dist-test-slave-r7ph"
server_key: "e59cbe715cbf2c9f5b47e841e5ea8498"
server_key_iv: "bfcb6f15f44a2b07d77fa129c3330747"
server_key_version: "encryptionkey@0"
I20260929 17:57:44.918128 9107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:44.943866 9107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:44.947921 9107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:44.948185 9107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:44.949880 9107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:57:44.949976 9107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:44.950057 9107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:57:44.950100 9107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:44.961773 9107 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:34861
I20260929 17:57:44.961864 9237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:34861 every 8 connection(s)
Sep 29 17:57:44 dist-test-slave-r7ph krb5kdc[8710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704664, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:57:44.962926 9107 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 17:57:44.967954 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:44.950966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:48555 (local address 127.7.118.62:33793)
0929 17:57:44.951151 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:44.951156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:44.952302 (+ 1146us) server_negotiation.cc:407] Connection header received
0929 17:57:44.953909 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:44.953915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:44.954000 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:44.954073 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:44.956557 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:44.957566 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:44.959413 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:44.959854 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:44.960005 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:44.963635 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:44.963640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:44.963643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:44.963682 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:44.965461 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:44.966140 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:44.966144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:44.966145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:44.966241 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:44.966709 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:44.966712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:44.966714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:44.966926 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:44.967040 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:44.967607 (+ 567us) server_negotiation.cc:299] Negotiation successful
0929 17:57:44.967760 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 17:57:44.969321 9131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:44.951558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33793 (local address 127.7.118.3:48555)
0929 17:57:44.952099 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:44.952168 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:44.953541 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:44.954674 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:44.954686 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:44.955423 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:44.956326 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:44.956350 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:44.957759 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:44.957765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:44.959177 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:44.959191 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:44.960193 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:44.960197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:44.960460 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:44.961239 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:44.961286 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:44.963341 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:44.965637 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:44.965645 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:44.965668 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:44.965988 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:44.966418 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:44.966421 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:44.966422 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:44.966593 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:44.967040 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:44.967049 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:44.967345 (+ 296us) client_negotiation.cc:769] Sending connection context
0929 17:57:44.967549 (+ 204us) client_negotiation.cc:240] Negotiation successful
0929 17:57:44.967936 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":161,"threads_started":1}
I20260929 17:57:44.969668 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 9107
I20260929 17:57:44.969875 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance
I20260929 17:57:44.970233 7640 external_mini_cluster.cc:1428] Setting key cfb6945b769506b5716dc26bcfc0aeb2
I20260929 17:57:44.971464 9238 heartbeater.cc:347] Connected to a master server at 127.7.118.62:33793
I20260929 17:57:44.971681 9238 heartbeater.cc:464] Registering TS with master...
I20260929 17:57:44.972316 9238 heartbeater.cc:511] Master 127.7.118.62:33793 requested a full tablet report, sending...
I20260929 17:57:44.974756 8755 ts_manager.cc:194] Registered new tserver with Master: 7df4a7cb612e48848f68934bcc690309 (127.7.118.3:34861)
I20260929 17:57:44.975558 8755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.3@KRBTEST.COM'} at 127.7.118.3:48555
I20260929 17:57:44.975827 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:57:44.984009 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:44.975460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:46991 (local address 127.7.118.62:33793)
0929 17:57:44.975582 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:44.975586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:44.975601 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:57:44.975659 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:44.975665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:44.975713 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:44.975787 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:44.976600 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:44.977619 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:44.978814 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:44.979063 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:44.979187 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:44.980449 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:44.980453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:44.980456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:44.980494 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:44.982106 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:44.982670 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:44.982674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:44.982676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:44.982774 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:44.983065 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:44.983069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:44.983071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:44.983284 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:44.983421 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:44.983713 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 17:57:44.983829 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 17:57:44.984020 9244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:44.975053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33793 (local address 127.7.118.3:46991)
0929 17:57:44.975367 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:44.975384 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:44.975486 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:44.975811 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:44.975815 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:44.976071 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:44.976416 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:44.976431 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:44.977813 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:44.977816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:44.978666 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:44.978683 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:44.979195 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:44.979199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:44.979293 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:44.979904 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:44.979918 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:44.980269 (+ 351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:44.982288 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:44.982292 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:44.982295 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:44.982550 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:44.982876 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:44.982879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:44.982883 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:44.982968 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:44.983406 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:44.983409 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:44.983517 (+ 108us) client_negotiation.cc:769] Sending connection context
0929 17:57:44.983655 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 17:57:44.983834 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":125,"threads_started":1}
Sep 29 17:57:44 dist-test-slave-r7ph krb5kdc[8710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704664, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 17:57:45 dist-test-slave-r7ph krb5kdc[8710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704664, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:57:45.008440 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:44.996711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57422 (local address 127.7.118.62:33793)
0929 17:57:44.997057 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:44.997061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:44.997076 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:57:44.997121 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:44.997124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:44.997164 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:44.997239 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:44.998349 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:44.998917 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.000145 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.000431 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.000546 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:45.003758 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:45.003762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:45.003764 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:45.003801 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:45.005855 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:45.006621 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:45.006625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:45.006626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:45.006727 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:45.007288 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:45.007292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:45.007294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:45.007527 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:45.007665 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:45.008030 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 17:57:45.008209 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258}
I20260929 17:57:45.012351 8755 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:57:45.015755 8755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:57:45.034189 9260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:45.028225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57044 (local address 127.7.118.1:42559)
0929 17:57:45.029098 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:45.029104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:45.029122 (+ 18us) server_negotiation.cc:407] Connection header received
0929 17:57:45.029188 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:45.029193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:45.029411 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:45.029711 (+ 300us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:45.030108 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.031256 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.032758 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.033389 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.033529 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:45.033705 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:45.033951 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 17:57:45.034066 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":113,"threads_started":1}
I20260929 17:57:45.034157 9255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:45.027136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:42559 (local address 127.0.0.1:57044)
0929 17:57:45.028225 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:45.028241 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:45.028347 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:45.029574 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:45.029577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:45.029608 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:45.029922 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:45.029930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.031439 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.031443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:45.032632 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:45.032642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.033533 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.033537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:45.033652 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:45.033692 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 17:57:45.033826 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 17:57:45.033978 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1019,"mutex_wait_us":186,"spinlock_wait_cycles":4480,"thread_start_us":247,"threads_started":1}
I20260929 17:57:45.034404 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:45.027894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:43843 (local address 127.0.0.1:46610)
0929 17:57:45.028586 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:45.028600 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:45.028682 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:45.029574 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:45.029577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:45.029607 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:45.029776 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:45.029784 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.030969 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.030972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:45.032656 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:45.032665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.033809 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.033812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:45.033957 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:45.033987 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 17:57:45.034056 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 17:57:45.034117 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":256,"spinlock_wait_cycles":5888,"thread_start_us":114,"threads_started":1}
I20260929 17:57:45.034816 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:45.028112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46610 (local address 127.7.118.2:43843)
0929 17:57:45.029069 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:45.029083 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:45.029111 (+ 28us) server_negotiation.cc:407] Connection header received
0929 17:57:45.029188 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:45.029193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:45.029406 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:45.029683 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:45.029903 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.030776 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.032798 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.033653 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.034203 (+ 550us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:45.034469 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:45.034530 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 17:57:45.034672 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":846,"thread_start_us":113,"threads_started":1}
I20260929 17:57:45.035036 9257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:45.026890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:34861 (local address 127.0.0.1:54764)
0929 17:57:45.027795 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:45.027888 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:45.028121 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:45.029068 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:45.029074 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:45.029101 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:45.029239 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:45.029245 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.030884 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.030888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:45.032592 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:45.032606 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.033789 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.033795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:45.034736 (+ 941us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:45.034763 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 17:57:45.034833 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 17:57:45.034905 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":919,"thread_start_us":116,"threads_started":1}
I20260929 17:57:45.035557 9256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:45.026959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54764 (local address 127.7.118.3:34861)
0929 17:57:45.027572 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:45.027583 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:45.027987 (+ 404us) server_negotiation.cc:407] Connection header received
0929 17:57:45.028338 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:45.028440 (+ 102us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:45.028685 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:45.028833 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:45.029358 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.030746 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.032762 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.033653 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.033781 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:45.033965 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:45.035291 (+ 1326us) server_negotiation.cc:299] Negotiation successful
0929 17:57:45.035412 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":136,"threads_started":1}
I20260929 17:57:45.037901 8892 tablet_service.cc:1467] Processing CreateTablet for tablet da4d50cf959b443da553410aa33e26ca (DEFAULT_TABLE table=test-table [id=f613169d4a8149ebb1c429657b5ade71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:45.037901 9032 tablet_service.cc:1467] Processing CreateTablet for tablet da4d50cf959b443da553410aa33e26ca (DEFAULT_TABLE table=test-table [id=f613169d4a8149ebb1c429657b5ade71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:45.038213 9172 tablet_service.cc:1467] Processing CreateTablet for tablet da4d50cf959b443da553410aa33e26ca (DEFAULT_TABLE table=test-table [id=f613169d4a8149ebb1c429657b5ade71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:45.039232 9032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da4d50cf959b443da553410aa33e26ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:45.039232 8892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da4d50cf959b443da553410aa33e26ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:45.039377 9172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da4d50cf959b443da553410aa33e26ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:45.047288 9261 tablet_bootstrap.cc:492] T da4d50cf959b443da553410aa33e26ca P 7df4a7cb612e48848f68934bcc690309: Bootstrap starting.
I20260929 17:57:45.048466 9263 tablet_bootstrap.cc:492] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd: Bootstrap starting.
I20260929 17:57:45.049144 9262 tablet_bootstrap.cc:492] T da4d50cf959b443da553410aa33e26ca P 8e9d2970919a488da31c348bc8bf2d8e: Bootstrap starting.
I20260929 17:57:45.049973 9261 tablet_bootstrap.cc:654] T da4d50cf959b443da553410aa33e26ca P 7df4a7cb612e48848f68934bcc690309: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:45.051079 9261 log.cc:824] T da4d50cf959b443da553410aa33e26ca P 7df4a7cb612e48848f68934bcc690309: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:45.051185 9263 tablet_bootstrap.cc:654] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:45.051718 9262 tablet_bootstrap.cc:654] T da4d50cf959b443da553410aa33e26ca P 8e9d2970919a488da31c348bc8bf2d8e: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:45.052194 9263 log.cc:824] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:45.052733 9262 log.cc:824] T da4d50cf959b443da553410aa33e26ca P 8e9d2970919a488da31c348bc8bf2d8e: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:45.054024 9261 tablet_bootstrap.cc:492] T da4d50cf959b443da553410aa33e26ca P 7df4a7cb612e48848f68934bcc690309: No bootstrap required, opened a new log
I20260929 17:57:45.054287 9261 ts_tablet_manager.cc:1397] T da4d50cf959b443da553410aa33e26ca P 7df4a7cb612e48848f68934bcc690309: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260929 17:57:45.054509 9263 tablet_bootstrap.cc:492] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd: No bootstrap required, opened a new log
I20260929 17:57:45.054716 9263 ts_tablet_manager.cc:1397] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260929 17:57:45.054816 9262 tablet_bootstrap.cc:492] T da4d50cf959b443da553410aa33e26ca P 8e9d2970919a488da31c348bc8bf2d8e: No bootstrap required, opened a new log
I20260929 17:57:45.054994 9262 ts_tablet_manager.cc:1397] T da4d50cf959b443da553410aa33e26ca P 8e9d2970919a488da31c348bc8bf2d8e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260929 17:57:45.058089 9263 raft_consensus.cc:348] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } } peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } }
I20260929 17:57:45.058494 9263 raft_consensus.cc:374] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:45.058566 9263 raft_consensus.cc:729] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c2215bc64b14d40b64835326a0cebbd, State: Initialized, Role: FOLLOWER
I20260929 17:57:45.058897 9261 raft_consensus.cc:348] T da4d50cf959b443da553410aa33e26ca P 7df4a7cb612e48848f68934bcc690309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } } peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } }
I20260929 17:57:45.059185 9261 raft_consensus.cc:374] T da4d50cf959b443da553410aa33e26ca P 7df4a7cb612e48848f68934bcc690309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:45.059238 9261 raft_consensus.cc:729] T da4d50cf959b443da553410aa33e26ca P 7df4a7cb612e48848f68934bcc690309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df4a7cb612e48848f68934bcc690309, State: Initialized, Role: FOLLOWER
I20260929 17:57:45.059147 9263 consensus_queue.cc:260] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } } peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } }
I20260929 17:57:45.059736 9261 consensus_queue.cc:260] T da4d50cf959b443da553410aa33e26ca P 7df4a7cb612e48848f68934bcc690309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } } peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } }
I20260929 17:57:45.060142 9098 heartbeater.cc:503] Master 127.7.118.62:33793 was elected leader, sending a full tablet report...
I20260929 17:57:45.060009 9262 raft_consensus.cc:348] T da4d50cf959b443da553410aa33e26ca P 8e9d2970919a488da31c348bc8bf2d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } } peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } }
I20260929 17:57:45.060264 9262 raft_consensus.cc:374] T da4d50cf959b443da553410aa33e26ca P 8e9d2970919a488da31c348bc8bf2d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:45.060310 9262 raft_consensus.cc:729] T da4d50cf959b443da553410aa33e26ca P 8e9d2970919a488da31c348bc8bf2d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e9d2970919a488da31c348bc8bf2d8e, State: Initialized, Role: FOLLOWER
I20260929 17:57:45.060590 9263 ts_tablet_manager.cc:1428] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 17:57:45.061005 9261 ts_tablet_manager.cc:1428] T da4d50cf959b443da553410aa33e26ca P 7df4a7cb612e48848f68934bcc690309: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 17:57:45.060959 9262 consensus_queue.cc:260] T da4d50cf959b443da553410aa33e26ca P 8e9d2970919a488da31c348bc8bf2d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } } peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } }
I20260929 17:57:45.061448 9238 heartbeater.cc:503] Master 127.7.118.62:33793 was elected leader, sending a full tablet report...
I20260929 17:57:45.061905 9262 ts_tablet_manager.cc:1428] T da4d50cf959b443da553410aa33e26ca P 8e9d2970919a488da31c348bc8bf2d8e: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260929 17:57:45.062311 8958 heartbeater.cc:503] Master 127.7.118.62:33793 was elected leader, sending a full tablet report...
W20260929 17:57:45.191883 9099 tablet.cc:2369] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:57:45.204779 8959 tablet.cc:2369] T da4d50cf959b443da553410aa33e26ca P 8e9d2970919a488da31c348bc8bf2d8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:57:45.214097 9239 tablet.cc:2369] T da4d50cf959b443da553410aa33e26ca P 7df4a7cb612e48848f68934bcc690309: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:57:45.434329 9267 raft_consensus.cc:482] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:57:45.434574 9267 raft_consensus.cc:504] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } } peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } }
I20260929 17:57:45.435791 9267 leader_election.cc:288] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df4a7cb612e48848f68934bcc690309 (127.7.118.3:34861), 8e9d2970919a488da31c348bc8bf2d8e (127.7.118.1:42559)
I20260929 17:57:45.441923 9260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:45.436212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:37511 (local address 127.7.118.1:42559)
0929 17:57:45.436442 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:45.436447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:45.436547 (+ 100us) server_negotiation.cc:407] Connection header received
0929 17:57:45.436705 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:45.436709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:45.436766 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:45.436863 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:45.437419 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.438215 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.440347 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.441032 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.441171 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:45.441248 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:45.441687 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 17:57:45.441786 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260929 17:57:45.441946 9256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:45.436193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:47075 (local address 127.7.118.3:34861)
0929 17:57:45.436418 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:45.436423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:45.436538 (+ 115us) server_negotiation.cc:407] Connection header received
0929 17:57:45.436705 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:45.436709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:45.436821 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:45.436898 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:45.437419 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.438227 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.440347 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.441072 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.441180 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:45.441248 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:45.441687 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 17:57:45.441815 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260929 17:57:45.441936 9270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:45.436081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:42559 (local address 127.7.118.2:37511)
0929 17:57:45.436460 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:45.436475 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:45.436588 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:45.436952 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:45.436955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:45.436989 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:45.437241 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:45.437250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.438384 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.438389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:45.440179 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:45.440193 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.441234 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.441239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:45.441478 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:45.441534 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 17:57:45.441622 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 17:57:45.441731 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":445,"spinlock_wait_cycles":28032,"thread_start_us":90,"threads_started":1}
I20260929 17:57:45.441968 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:45.436072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:34861 (local address 127.7.118.2:47075)
0929 17:57:45.436416 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:45.436433 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:45.436588 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:45.436952 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:45.436955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:45.436989 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:45.437241 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:45.437250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.438384 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.438389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:45.440183 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:45.440193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.441274 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.441277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:45.441474 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:45.441530 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 17:57:45.441622 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 17:57:45.441731 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":245,"spinlock_wait_cycles":23808,"thread_start_us":74,"threads_started":1}
I20260929 17:57:45.442898 9192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da4d50cf959b443da553410aa33e26ca" candidate_uuid: "2c2215bc64b14d40b64835326a0cebbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df4a7cb612e48848f68934bcc690309" is_pre_election: true
I20260929 17:57:45.442906 8912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da4d50cf959b443da553410aa33e26ca" candidate_uuid: "2c2215bc64b14d40b64835326a0cebbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e9d2970919a488da31c348bc8bf2d8e" is_pre_election: true
I20260929 17:57:45.443310 9192 raft_consensus.cc:2454] T da4d50cf959b443da553410aa33e26ca P 7df4a7cb612e48848f68934bcc690309 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c2215bc64b14d40b64835326a0cebbd in term 0.
I20260929 17:57:45.443333 8912 raft_consensus.cc:2454] T da4d50cf959b443da553410aa33e26ca P 8e9d2970919a488da31c348bc8bf2d8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c2215bc64b14d40b64835326a0cebbd in term 0.
I20260929 17:57:45.443904 8983 leader_election.cc:302] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2c2215bc64b14d40b64835326a0cebbd, 7df4a7cb612e48848f68934bcc690309, 8e9d2970919a488da31c348bc8bf2d8e; no voters:
I20260929 17:57:45.444242 9267 raft_consensus.cc:2781] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:57:45.444340 9267 raft_consensus.cc:482] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:57:45.444399 9267 raft_consensus.cc:3037] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:45.445571 9267 raft_consensus.cc:504] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } } peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } }
I20260929 17:57:45.446003 9267 leader_election.cc:288] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd [CANDIDATE]: Term 1 election: Requested vote from peers 7df4a7cb612e48848f68934bcc690309 (127.7.118.3:34861), 8e9d2970919a488da31c348bc8bf2d8e (127.7.118.1:42559)
I20260929 17:57:45.446525 9192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da4d50cf959b443da553410aa33e26ca" candidate_uuid: "2c2215bc64b14d40b64835326a0cebbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df4a7cb612e48848f68934bcc690309"
I20260929 17:57:45.446666 9192 raft_consensus.cc:3037] T da4d50cf959b443da553410aa33e26ca P 7df4a7cb612e48848f68934bcc690309 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:45.446666 8912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da4d50cf959b443da553410aa33e26ca" candidate_uuid: "2c2215bc64b14d40b64835326a0cebbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e9d2970919a488da31c348bc8bf2d8e"
I20260929 17:57:45.446870 8912 raft_consensus.cc:3037] T da4d50cf959b443da553410aa33e26ca P 8e9d2970919a488da31c348bc8bf2d8e [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:45.447937 9192 raft_consensus.cc:2454] T da4d50cf959b443da553410aa33e26ca P 7df4a7cb612e48848f68934bcc690309 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c2215bc64b14d40b64835326a0cebbd in term 1.
I20260929 17:57:45.447968 8912 raft_consensus.cc:2454] T da4d50cf959b443da553410aa33e26ca P 8e9d2970919a488da31c348bc8bf2d8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c2215bc64b14d40b64835326a0cebbd in term 1.
I20260929 17:57:45.448482 8983 leader_election.cc:302] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c2215bc64b14d40b64835326a0cebbd, 7df4a7cb612e48848f68934bcc690309; no voters:
I20260929 17:57:45.448690 9267 raft_consensus.cc:2781] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:45.448935 9267 raft_consensus.cc:686] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd [term 1 LEADER]: Becoming Leader. State: Replica: 2c2215bc64b14d40b64835326a0cebbd, State: Running, Role: LEADER
I20260929 17:57:45.449312 9267 consensus_queue.cc:237] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } } peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } }
I20260929 17:57:45.453105 8754 catalog_manager.cc:5495] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c2215bc64b14d40b64835326a0cebbd (127.7.118.2). New cstate: current_term: 1 leader_uuid: "2c2215bc64b14d40b64835326a0cebbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } health_report { overall_health: HEALTHY } } }
I20260929 17:57:45.499351 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:45.493584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46616 (local address 127.7.118.2:43843)
0929 17:57:45.493724 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:45.493729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:45.493839 (+ 110us) server_negotiation.cc:407] Connection header received
0929 17:57:45.494158 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:45.494162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:45.494307 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:45.494443 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:57:45.494822 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.495759 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.497317 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.497751 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.497884 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:45.498369 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:57:45.498898 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:57:45.499040 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:45.499124 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 17:57:45.499192 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 17:57:45.506110 9192 raft_consensus.cc:1261] T da4d50cf959b443da553410aa33e26ca P 7df4a7cb612e48848f68934bcc690309 [term 1 FOLLOWER]: Refusing update from remote peer 2c2215bc64b14d40b64835326a0cebbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:45.506111 8912 raft_consensus.cc:1261] T da4d50cf959b443da553410aa33e26ca P 8e9d2970919a488da31c348bc8bf2d8e [term 1 FOLLOWER]: Refusing update from remote peer 2c2215bc64b14d40b64835326a0cebbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:45.506959 9272 consensus_queue.cc:1035] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:57:45.507200 9267 consensus_queue.cc:1035] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:57:45.515329 9275 mvcc.cc:205] Tried to move back new op lower bound from 7334726309905321984 to 7334726309683294208. Current Snapshot: MvccSnapshot[applied={T|T < 7334726309905321984}]
I20260929 17:57:45.516287 9277 mvcc.cc:205] Tried to move back new op lower bound from 7334726309905321984 to 7334726309683294208. Current Snapshot: MvccSnapshot[applied={T|T < 7334726309905321984}]
I20260929 17:57:45.519322 9276 mvcc.cc:205] Tried to move back new op lower bound from 7334726309905321984 to 7334726309683294208. Current Snapshot: MvccSnapshot[applied={T|T < 7334726309905321984}]
I20260929 17:57:45.528317 8754 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:57:45.528501 8754 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:57:45.531486 8754 catalog_manager.cc:5782] T 00000000000000000000000000000000 P acb652fa8bad4fc99cc18238f6297523: Sending DeleteTablet for 3 replicas of tablet da4d50cf959b443da553410aa33e26ca
I20260929 17:57:45.532692 8892 tablet_service.cc:1514] Processing DeleteTablet for tablet da4d50cf959b443da553410aa33e26ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:45 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:57044
I20260929 17:57:45.532625 9172 tablet_service.cc:1514] Processing DeleteTablet for tablet da4d50cf959b443da553410aa33e26ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:45 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:54764
I20260929 17:57:45.532796 9032 tablet_service.cc:1514] Processing DeleteTablet for tablet da4d50cf959b443da553410aa33e26ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:45 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:46610
I20260929 17:57:45.533252 9285 tablet_replica.cc:330] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd: stopping tablet replica
I20260929 17:57:45.533485 9284 tablet_replica.cc:330] T da4d50cf959b443da553410aa33e26ca P 7df4a7cb612e48848f68934bcc690309: stopping tablet replica
I20260929 17:57:45.533653 9285 raft_consensus.cc:2229] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:57:45.533881 9284 raft_consensus.cc:2229] T da4d50cf959b443da553410aa33e26ca P 7df4a7cb612e48848f68934bcc690309 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:57:45.534142 9285 raft_consensus.cc:2258] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:57:45.534317 9284 raft_consensus.cc:2258] T da4d50cf959b443da553410aa33e26ca P 7df4a7cb612e48848f68934bcc690309 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:57:45.534286 8754 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:57:45.534811 8754 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:57:45.535379 9286 tablet_replica.cc:330] T da4d50cf959b443da553410aa33e26ca P 8e9d2970919a488da31c348bc8bf2d8e: stopping tablet replica
I20260929 17:57:45.535935 9286 raft_consensus.cc:2229] T da4d50cf959b443da553410aa33e26ca P 8e9d2970919a488da31c348bc8bf2d8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:57:45.535977 9285 ts_tablet_manager.cc:1905] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:57:45.537312 9284 ts_tablet_manager.cc:1905] T da4d50cf959b443da553410aa33e26ca P 7df4a7cb612e48848f68934bcc690309: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:57:45.538708 9286 raft_consensus.cc:2258] T da4d50cf959b443da553410aa33e26ca P 8e9d2970919a488da31c348bc8bf2d8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:57:45.540407 9284 ts_tablet_manager.cc:1918] T da4d50cf959b443da553410aa33e26ca P 7df4a7cb612e48848f68934bcc690309: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:57:45.540588 9284 log.cc:1196] T da4d50cf959b443da553410aa33e26ca P 7df4a7cb612e48848f68934bcc690309: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-2/wal/wals/da4d50cf959b443da553410aa33e26ca
I20260929 17:57:45.541038 8892 tablet_service.cc:1467] Processing CreateTablet for tablet 779d20f65f0441309692d77a92b36d88 (DEFAULT_TABLE table=test-table [id=67f26a184a8843bb93ca2f32db942b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:45.541036 9172 tablet_service.cc:1467] Processing CreateTablet for tablet 779d20f65f0441309692d77a92b36d88 (DEFAULT_TABLE table=test-table [id=67f26a184a8843bb93ca2f32db942b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:45.541180 9286 ts_tablet_manager.cc:1905] T da4d50cf959b443da553410aa33e26ca P 8e9d2970919a488da31c348bc8bf2d8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:57:45.541306 9172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 779d20f65f0441309692d77a92b36d88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:45.541318 8892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 779d20f65f0441309692d77a92b36d88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:45.541585 9032 tablet_service.cc:1467] Processing CreateTablet for tablet 779d20f65f0441309692d77a92b36d88 (DEFAULT_TABLE table=test-table [id=67f26a184a8843bb93ca2f32db942b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:45.541843 9285 ts_tablet_manager.cc:1918] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:57:45.541869 9032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 779d20f65f0441309692d77a92b36d88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:45.543833 9261 tablet_bootstrap.cc:492] T 779d20f65f0441309692d77a92b36d88 P 7df4a7cb612e48848f68934bcc690309: Bootstrap starting.
I20260929 17:57:45.544332 9284 ts_tablet_manager.cc:1939] T da4d50cf959b443da553410aa33e26ca P 7df4a7cb612e48848f68934bcc690309: Deleting consensus metadata
I20260929 17:57:45.544646 9263 tablet_bootstrap.cc:492] T 779d20f65f0441309692d77a92b36d88 P 2c2215bc64b14d40b64835326a0cebbd: Bootstrap starting.
I20260929 17:57:45.544782 9286 ts_tablet_manager.cc:1918] T da4d50cf959b443da553410aa33e26ca P 8e9d2970919a488da31c348bc8bf2d8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:57:45.544909 9286 log.cc:1196] T da4d50cf959b443da553410aa33e26ca P 8e9d2970919a488da31c348bc8bf2d8e: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-0/wal/wals/da4d50cf959b443da553410aa33e26ca
I20260929 17:57:45.545265 9261 tablet_bootstrap.cc:654] T 779d20f65f0441309692d77a92b36d88 P 7df4a7cb612e48848f68934bcc690309: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:45.545282 9286 ts_tablet_manager.cc:1939] T da4d50cf959b443da553410aa33e26ca P 8e9d2970919a488da31c348bc8bf2d8e: Deleting consensus metadata
I20260929 17:57:45.545804 9263 tablet_bootstrap.cc:654] T 779d20f65f0441309692d77a92b36d88 P 2c2215bc64b14d40b64835326a0cebbd: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:45.545919 9262 tablet_bootstrap.cc:492] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e: Bootstrap starting.
I20260929 17:57:45.546615 9285 log.cc:1196] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-1/wal/wals/da4d50cf959b443da553410aa33e26ca
I20260929 17:57:45.546967 9285 ts_tablet_manager.cc:1939] T da4d50cf959b443da553410aa33e26ca P 2c2215bc64b14d40b64835326a0cebbd: Deleting consensus metadata
I20260929 17:57:45.547106 8739 catalog_manager.cc:4841] TS 7df4a7cb612e48848f68934bcc690309 (127.7.118.3:34861): tablet da4d50cf959b443da553410aa33e26ca (table test-table [id=f613169d4a8149ebb1c429657b5ade71]) successfully deleted
I20260929 17:57:45.547394 9262 tablet_bootstrap.cc:654] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:45.548064 8742 catalog_manager.cc:4841] TS 8e9d2970919a488da31c348bc8bf2d8e (127.7.118.1:42559): tablet da4d50cf959b443da553410aa33e26ca (table test-table [id=f613169d4a8149ebb1c429657b5ade71]) successfully deleted
I20260929 17:57:45.548725 9261 tablet_bootstrap.cc:492] T 779d20f65f0441309692d77a92b36d88 P 7df4a7cb612e48848f68934bcc690309: No bootstrap required, opened a new log
I20260929 17:57:45.548864 9263 tablet_bootstrap.cc:492] T 779d20f65f0441309692d77a92b36d88 P 2c2215bc64b14d40b64835326a0cebbd: No bootstrap required, opened a new log
I20260929 17:57:45.548877 9261 ts_tablet_manager.cc:1397] T 779d20f65f0441309692d77a92b36d88 P 7df4a7cb612e48848f68934bcc690309: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260929 17:57:45.548959 9263 ts_tablet_manager.cc:1397] T 779d20f65f0441309692d77a92b36d88 P 2c2215bc64b14d40b64835326a0cebbd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 17:57:45.549031 8742 catalog_manager.cc:4841] TS 2c2215bc64b14d40b64835326a0cebbd (127.7.118.2:43843): tablet da4d50cf959b443da553410aa33e26ca (table test-table [id=f613169d4a8149ebb1c429657b5ade71]) successfully deleted
I20260929 17:57:45.549443 9263 raft_consensus.cc:348] T 779d20f65f0441309692d77a92b36d88 P 2c2215bc64b14d40b64835326a0cebbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } }
I20260929 17:57:45.549584 9263 raft_consensus.cc:374] T 779d20f65f0441309692d77a92b36d88 P 2c2215bc64b14d40b64835326a0cebbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:45.549638 9263 raft_consensus.cc:729] T 779d20f65f0441309692d77a92b36d88 P 2c2215bc64b14d40b64835326a0cebbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c2215bc64b14d40b64835326a0cebbd, State: Initialized, Role: FOLLOWER
I20260929 17:57:45.549642 9262 tablet_bootstrap.cc:492] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e: No bootstrap required, opened a new log
I20260929 17:57:45.549721 9262 ts_tablet_manager.cc:1397] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 17:57:45.549809 9263 consensus_queue.cc:260] T 779d20f65f0441309692d77a92b36d88 P 2c2215bc64b14d40b64835326a0cebbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } }
I20260929 17:57:45.550166 9262 raft_consensus.cc:348] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } }
I20260929 17:57:45.550312 9262 raft_consensus.cc:374] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:45.550346 9262 raft_consensus.cc:729] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e9d2970919a488da31c348bc8bf2d8e, State: Initialized, Role: FOLLOWER
I20260929 17:57:45.550570 9262 consensus_queue.cc:260] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } }
I20260929 17:57:45.550864 9262 ts_tablet_manager.cc:1428] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:57:45.550841 9261 raft_consensus.cc:348] T 779d20f65f0441309692d77a92b36d88 P 7df4a7cb612e48848f68934bcc690309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } }
I20260929 17:57:45.551011 9261 raft_consensus.cc:374] T 779d20f65f0441309692d77a92b36d88 P 7df4a7cb612e48848f68934bcc690309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:45.551100 9261 raft_consensus.cc:729] T 779d20f65f0441309692d77a92b36d88 P 7df4a7cb612e48848f68934bcc690309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df4a7cb612e48848f68934bcc690309, State: Initialized, Role: FOLLOWER
I20260929 17:57:45.551334 9263 ts_tablet_manager.cc:1428] T 779d20f65f0441309692d77a92b36d88 P 2c2215bc64b14d40b64835326a0cebbd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 17:57:45.551319 9261 consensus_queue.cc:260] T 779d20f65f0441309692d77a92b36d88 P 7df4a7cb612e48848f68934bcc690309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } }
I20260929 17:57:45.551661 9261 ts_tablet_manager.cc:1428] T 779d20f65f0441309692d77a92b36d88 P 7df4a7cb612e48848f68934bcc690309: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260929 17:57:45.620209 9269 raft_consensus.cc:482] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:57:45.620410 9269 raft_consensus.cc:504] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } }
I20260929 17:57:45.621826 9269 leader_election.cc:288] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c2215bc64b14d40b64835326a0cebbd (127.7.118.2:43843), 7df4a7cb612e48848f68934bcc690309 (127.7.118.3:34861)
I20260929 17:57:45.627967 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:45.622284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:43843 (local address 127.7.118.1:37053)
0929 17:57:45.622756 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:45.622772 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:45.622925 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:45.623585 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:45.623589 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:45.623611 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:45.623861 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:45.623869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.624876 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.624880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:45.626240 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:45.626280 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.627413 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.627416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:45.627517 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:45.627541 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 17:57:45.627670 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 17:57:45.627777 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":106,"thread_start_us":74,"threads_started":1}
I20260929 17:57:45.627960 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:45.622419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:37053 (local address 127.7.118.2:43843)
0929 17:57:45.622570 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:45.622575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:45.622871 (+ 296us) server_negotiation.cc:407] Connection header received
0929 17:57:45.623339 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:45.623342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:45.623388 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:45.623463 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:45.623982 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.624474 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.626421 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.627177 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.627345 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:45.627508 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:45.627690 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 17:57:45.627784 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 17:57:45.629043 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:45.622284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:34861 (local address 127.7.118.1:58355)
0929 17:57:45.622756 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:45.622772 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:45.622919 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:45.623279 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:45.623282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:45.623304 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:45.623605 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:45.623615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.624656 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.624662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:45.626240 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:45.626280 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.628384 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.628388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:45.628679 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:45.628706 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 17:57:45.628791 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 17:57:45.628881 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":289,"thread_start_us":109,"threads_started":1}
I20260929 17:57:45.629076 9256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:45.622426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:58355 (local address 127.7.118.3:34861)
0929 17:57:45.622646 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:45.622650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:45.622912 (+ 262us) server_negotiation.cc:407] Connection header received
0929 17:57:45.623034 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:45.623038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:45.623090 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:45.623201 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:45.623776 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.624559 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.627599 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.628233 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.628359 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:45.628423 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:45.628833 (+ 410us) server_negotiation.cc:299] Negotiation successful
0929 17:57:45.628928 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260929 17:57:45.628986 9052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779d20f65f0441309692d77a92b36d88" candidate_uuid: "8e9d2970919a488da31c348bc8bf2d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c2215bc64b14d40b64835326a0cebbd" is_pre_election: true
I20260929 17:57:45.629274 9052 raft_consensus.cc:2454] T 779d20f65f0441309692d77a92b36d88 P 2c2215bc64b14d40b64835326a0cebbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e9d2970919a488da31c348bc8bf2d8e in term 0.
I20260929 17:57:45.629758 9192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779d20f65f0441309692d77a92b36d88" candidate_uuid: "8e9d2970919a488da31c348bc8bf2d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df4a7cb612e48848f68934bcc690309" is_pre_election: true
I20260929 17:57:45.629837 8846 leader_election.cc:302] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c2215bc64b14d40b64835326a0cebbd, 8e9d2970919a488da31c348bc8bf2d8e; no voters:
I20260929 17:57:45.629914 9192 raft_consensus.cc:2454] T 779d20f65f0441309692d77a92b36d88 P 7df4a7cb612e48848f68934bcc690309 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e9d2970919a488da31c348bc8bf2d8e in term 0.
I20260929 17:57:45.630158 9269 raft_consensus.cc:2781] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:57:45.630246 9269 raft_consensus.cc:482] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:57:45.630322 9269 raft_consensus.cc:3037] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:45.631731 9269 raft_consensus.cc:504] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } }
I20260929 17:57:45.632171 9269 leader_election.cc:288] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e [CANDIDATE]: Term 1 election: Requested vote from peers 2c2215bc64b14d40b64835326a0cebbd (127.7.118.2:43843), 7df4a7cb612e48848f68934bcc690309 (127.7.118.3:34861)
I20260929 17:57:45.632534 9052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779d20f65f0441309692d77a92b36d88" candidate_uuid: "8e9d2970919a488da31c348bc8bf2d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c2215bc64b14d40b64835326a0cebbd"
I20260929 17:57:45.632572 9192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779d20f65f0441309692d77a92b36d88" candidate_uuid: "8e9d2970919a488da31c348bc8bf2d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df4a7cb612e48848f68934bcc690309"
I20260929 17:57:45.632686 9052 raft_consensus.cc:3037] T 779d20f65f0441309692d77a92b36d88 P 2c2215bc64b14d40b64835326a0cebbd [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:45.632696 9192 raft_consensus.cc:3037] T 779d20f65f0441309692d77a92b36d88 P 7df4a7cb612e48848f68934bcc690309 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:45.633818 9052 raft_consensus.cc:2454] T 779d20f65f0441309692d77a92b36d88 P 2c2215bc64b14d40b64835326a0cebbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e9d2970919a488da31c348bc8bf2d8e in term 1.
I20260929 17:57:45.634042 9192 raft_consensus.cc:2454] T 779d20f65f0441309692d77a92b36d88 P 7df4a7cb612e48848f68934bcc690309 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e9d2970919a488da31c348bc8bf2d8e in term 1.
I20260929 17:57:45.634569 8846 leader_election.cc:302] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c2215bc64b14d40b64835326a0cebbd, 8e9d2970919a488da31c348bc8bf2d8e; no voters:
I20260929 17:57:45.634819 9269 raft_consensus.cc:2781] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:45.635095 9269 raft_consensus.cc:686] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e [term 1 LEADER]: Becoming Leader. State: Replica: 8e9d2970919a488da31c348bc8bf2d8e, State: Running, Role: LEADER
I20260929 17:57:45.635704 9269 consensus_queue.cc:237] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } }
I20260929 17:57:45.639001 8755 catalog_manager.cc:5495] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e9d2970919a488da31c348bc8bf2d8e (127.7.118.1). New cstate: current_term: 1 leader_uuid: "8e9d2970919a488da31c348bc8bf2d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } health_report { overall_health: UNKNOWN } } }
Sep 29 17:57:45 dist-test-slave-r7ph krb5kdc[8710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704664, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:57:45.687256 9300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:45.677142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57434 (local address 127.7.118.62:33793)
0929 17:57:45.677391 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:45.677400 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:45.677418 (+ 18us) server_negotiation.cc:407] Connection header received
0929 17:57:45.677488 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:45.677491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:45.677538 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:45.677651 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:45.678463 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.679053 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.679958 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.680198 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.680326 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:45.683002 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:45.683006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:45.683009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:45.683039 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:45.685131 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:45.685688 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:45.685693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:45.685696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:45.685786 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:45.686085 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:45.686089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:45.686091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:45.686380 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:45.686497 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:45.686818 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 17:57:45.687069 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":134,"server-negotiator.queue_time_us":187,"thread_start_us":109,"threads_started":1}
I20260929 17:57:45.687256 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:45.676876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33793 (local address 127.0.0.1:57434)
0929 17:57:45.677233 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:45.677252 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:45.677384 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:45.677669 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:45.677672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:45.678152 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:45.678318 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:45.678330 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.679174 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.679180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:45.679825 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:45.679834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.680329 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.680333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:45.680411 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:45.681165 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:45.681179 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:45.682816 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:45.685266 (+ 2450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:45.685281 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:45.685290 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:45.685572 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:45.685888 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:45.685893 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:45.685896 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:45.685981 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:45.686497 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:45.686507 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:45.686629 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 17:57:45.686757 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 17:57:45.687007 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":1,"thread_start_us":158,"threads_started":1}
I20260929 17:57:45.692623 8755 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:57434:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 17:57:45.693475 8755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:57:45.699810 8892 tablet_service.cc:1467] Processing CreateTablet for tablet 38791dcade5c442a8945100a6589624c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7a3c7c3d67e942a284561d5f261cacfb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 17:57:45.700230 8892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38791dcade5c442a8945100a6589624c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:45.700165 9032 tablet_service.cc:1467] Processing CreateTablet for tablet 38791dcade5c442a8945100a6589624c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7a3c7c3d67e942a284561d5f261cacfb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 17:57:45.700542 9032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38791dcade5c442a8945100a6589624c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:45.702848 9172 tablet_service.cc:1467] Processing CreateTablet for tablet 38791dcade5c442a8945100a6589624c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7a3c7c3d67e942a284561d5f261cacfb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 17:57:45.703253 9172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38791dcade5c442a8945100a6589624c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:45.706475 9262 tablet_bootstrap.cc:492] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e: Bootstrap starting.
I20260929 17:57:45.707510 9262 tablet_bootstrap.cc:654] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:45.709924 9263 tablet_bootstrap.cc:492] T 38791dcade5c442a8945100a6589624c P 2c2215bc64b14d40b64835326a0cebbd: Bootstrap starting.
I20260929 17:57:45.710479 9262 tablet_bootstrap.cc:492] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e: No bootstrap required, opened a new log
I20260929 17:57:45.710572 9262 ts_tablet_manager.cc:1397] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 17:57:45.710805 9263 tablet_bootstrap.cc:654] T 38791dcade5c442a8945100a6589624c P 2c2215bc64b14d40b64835326a0cebbd: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:45.711052 9262 raft_consensus.cc:348] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } }
I20260929 17:57:45.711169 9262 raft_consensus.cc:374] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:45.711199 9262 raft_consensus.cc:729] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e9d2970919a488da31c348bc8bf2d8e, State: Initialized, Role: FOLLOWER
I20260929 17:57:45.711339 9262 consensus_queue.cc:260] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } }
I20260929 17:57:45.711620 9269 tablet_replica.cc:439] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } } }
I20260929 17:57:45.711781 9269 tablet_replica.cc:442] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:57:45.711634 9262 ts_tablet_manager.cc:1428] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:57:45.712330 9261 tablet_bootstrap.cc:492] T 38791dcade5c442a8945100a6589624c P 7df4a7cb612e48848f68934bcc690309: Bootstrap starting.
I20260929 17:57:45.713409 9261 tablet_bootstrap.cc:654] T 38791dcade5c442a8945100a6589624c P 7df4a7cb612e48848f68934bcc690309: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:45.715128 9261 tablet_bootstrap.cc:492] T 38791dcade5c442a8945100a6589624c P 7df4a7cb612e48848f68934bcc690309: No bootstrap required, opened a new log
I20260929 17:57:45.715241 9261 ts_tablet_manager.cc:1397] T 38791dcade5c442a8945100a6589624c P 7df4a7cb612e48848f68934bcc690309: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 17:57:45.715747 9261 raft_consensus.cc:348] T 38791dcade5c442a8945100a6589624c P 7df4a7cb612e48848f68934bcc690309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } }
I20260929 17:57:45.715889 9261 raft_consensus.cc:374] T 38791dcade5c442a8945100a6589624c P 7df4a7cb612e48848f68934bcc690309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:45.715926 9261 raft_consensus.cc:729] T 38791dcade5c442a8945100a6589624c P 7df4a7cb612e48848f68934bcc690309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df4a7cb612e48848f68934bcc690309, State: Initialized, Role: FOLLOWER
I20260929 17:57:45.716059 9261 consensus_queue.cc:260] T 38791dcade5c442a8945100a6589624c P 7df4a7cb612e48848f68934bcc690309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } }
I20260929 17:57:45.716389 9261 ts_tablet_manager.cc:1428] T 38791dcade5c442a8945100a6589624c P 7df4a7cb612e48848f68934bcc690309: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260929 17:57:45.716657 9268 tablet_replica.cc:439] T 38791dcade5c442a8945100a6589624c P 7df4a7cb612e48848f68934bcc690309: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } } }
I20260929 17:57:45.716813 9268 tablet_replica.cc:442] T 38791dcade5c442a8945100a6589624c P 7df4a7cb612e48848f68934bcc690309: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:57:45.718415 9263 tablet_bootstrap.cc:492] T 38791dcade5c442a8945100a6589624c P 2c2215bc64b14d40b64835326a0cebbd: No bootstrap required, opened a new log
I20260929 17:57:45.718525 9263 ts_tablet_manager.cc:1397] T 38791dcade5c442a8945100a6589624c P 2c2215bc64b14d40b64835326a0cebbd: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260929 17:57:45.718936 9263 raft_consensus.cc:348] T 38791dcade5c442a8945100a6589624c P 2c2215bc64b14d40b64835326a0cebbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } }
I20260929 17:57:45.719058 9263 raft_consensus.cc:374] T 38791dcade5c442a8945100a6589624c P 2c2215bc64b14d40b64835326a0cebbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:45.719098 9263 raft_consensus.cc:729] T 38791dcade5c442a8945100a6589624c P 2c2215bc64b14d40b64835326a0cebbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c2215bc64b14d40b64835326a0cebbd, State: Initialized, Role: FOLLOWER
I20260929 17:57:45.719238 9263 consensus_queue.cc:260] T 38791dcade5c442a8945100a6589624c P 2c2215bc64b14d40b64835326a0cebbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } }
I20260929 17:57:45.719560 9263 ts_tablet_manager.cc:1428] T 38791dcade5c442a8945100a6589624c P 2c2215bc64b14d40b64835326a0cebbd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 17:57:45.719564 9267 tablet_replica.cc:439] T 38791dcade5c442a8945100a6589624c P 2c2215bc64b14d40b64835326a0cebbd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } } }
I20260929 17:57:45.719749 9267 tablet_replica.cc:442] T 38791dcade5c442a8945100a6589624c P 2c2215bc64b14d40b64835326a0cebbd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:57:45.904282 9269 raft_consensus.cc:482] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:57:45.904433 9269 raft_consensus.cc:504] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } }
I20260929 17:57:45.904997 9269 leader_election.cc:288] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c2215bc64b14d40b64835326a0cebbd (127.7.118.2:43843), 7df4a7cb612e48848f68934bcc690309 (127.7.118.3:34861)
I20260929 17:57:45.905587 9052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38791dcade5c442a8945100a6589624c" candidate_uuid: "8e9d2970919a488da31c348bc8bf2d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c2215bc64b14d40b64835326a0cebbd" is_pre_election: true
I20260929 17:57:45.905587 9192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38791dcade5c442a8945100a6589624c" candidate_uuid: "8e9d2970919a488da31c348bc8bf2d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df4a7cb612e48848f68934bcc690309" is_pre_election: true
I20260929 17:57:45.905761 9192 raft_consensus.cc:2454] T 38791dcade5c442a8945100a6589624c P 7df4a7cb612e48848f68934bcc690309 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e9d2970919a488da31c348bc8bf2d8e in term 0.
I20260929 17:57:45.905798 9052 raft_consensus.cc:2454] T 38791dcade5c442a8945100a6589624c P 2c2215bc64b14d40b64835326a0cebbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e9d2970919a488da31c348bc8bf2d8e in term 0.
I20260929 17:57:45.906174 8843 leader_election.cc:302] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2c2215bc64b14d40b64835326a0cebbd, 7df4a7cb612e48848f68934bcc690309, 8e9d2970919a488da31c348bc8bf2d8e; no voters:
I20260929 17:57:45.906461 9269 raft_consensus.cc:2781] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:57:45.906520 9269 raft_consensus.cc:482] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:57:45.906548 9269 raft_consensus.cc:3037] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:45.907630 9269 raft_consensus.cc:504] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } }
I20260929 17:57:45.908020 9269 leader_election.cc:288] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e [CANDIDATE]: Term 1 election: Requested vote from peers 2c2215bc64b14d40b64835326a0cebbd (127.7.118.2:43843), 7df4a7cb612e48848f68934bcc690309 (127.7.118.3:34861)
I20260929 17:57:45.908571 9052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38791dcade5c442a8945100a6589624c" candidate_uuid: "8e9d2970919a488da31c348bc8bf2d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c2215bc64b14d40b64835326a0cebbd"
I20260929 17:57:45.908571 9192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38791dcade5c442a8945100a6589624c" candidate_uuid: "8e9d2970919a488da31c348bc8bf2d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df4a7cb612e48848f68934bcc690309"
I20260929 17:57:45.908721 9192 raft_consensus.cc:3037] T 38791dcade5c442a8945100a6589624c P 7df4a7cb612e48848f68934bcc690309 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:45.908720 9052 raft_consensus.cc:3037] T 38791dcade5c442a8945100a6589624c P 2c2215bc64b14d40b64835326a0cebbd [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:45.909926 9192 raft_consensus.cc:2454] T 38791dcade5c442a8945100a6589624c P 7df4a7cb612e48848f68934bcc690309 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e9d2970919a488da31c348bc8bf2d8e in term 1.
I20260929 17:57:45.910022 9052 raft_consensus.cc:2454] T 38791dcade5c442a8945100a6589624c P 2c2215bc64b14d40b64835326a0cebbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e9d2970919a488da31c348bc8bf2d8e in term 1.
I20260929 17:57:45.910382 8846 leader_election.cc:302] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c2215bc64b14d40b64835326a0cebbd, 8e9d2970919a488da31c348bc8bf2d8e; no voters:
I20260929 17:57:45.910593 9269 raft_consensus.cc:2781] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:45.910717 9269 raft_consensus.cc:686] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e [term 1 LEADER]: Becoming Leader. State: Replica: 8e9d2970919a488da31c348bc8bf2d8e, State: Running, Role: LEADER
I20260929 17:57:45.910898 9269 consensus_queue.cc:237] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } }
I20260929 17:57:45.911645 9292 tablet_replica.cc:439] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e: TxnStatusTablet state changed. Reason: New leader 8e9d2970919a488da31c348bc8bf2d8e. Latest consensus state: current_term: 1 leader_uuid: "8e9d2970919a488da31c348bc8bf2d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } } }
I20260929 17:57:45.911837 9292 tablet_replica.cc:442] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e: This TxnStatusTablet replica's current role is: LEADER
I20260929 17:57:45.912150 9305 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 17:57:45.912348 9305 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 17:57:45.913024 8753 catalog_manager.cc:5495] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e9d2970919a488da31c348bc8bf2d8e (127.7.118.1). New cstate: current_term: 1 leader_uuid: "8e9d2970919a488da31c348bc8bf2d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } health_report { overall_health: UNKNOWN } } }
I20260929 17:57:45.956477 9308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:45.952224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57046 (local address 127.7.118.1:42559)
0929 17:57:45.952511 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:45.952515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:45.952532 (+ 17us) server_negotiation.cc:407] Connection header received
0929 17:57:45.952714 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:45.952717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:45.952836 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:45.952961 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:57:45.953233 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.953799 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.954790 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.955035 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.955150 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:45.955481 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:57:45.956003 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:57:45.956128 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:45.956253 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 17:57:45.956326 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":93,"threads_started":1}
I20260929 17:57:45.956485 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:45.952066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:42559 (local address 127.0.0.1:57046)
0929 17:57:45.952330 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:45.952350 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:45.952495 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:45.952986 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:45.952988 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:45.952997 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:57:45.953117 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:45.953122 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.953930 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.953933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:45.954658 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:45.954668 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.955157 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.955161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:45.955262 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:45.955314 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:57:45.956116 (+ 802us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:57:45.956138 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 17:57:45.956225 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 17:57:45.956330 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 17:57:45.959841 9192 raft_consensus.cc:1261] T 38791dcade5c442a8945100a6589624c P 7df4a7cb612e48848f68934bcc690309 [term 1 FOLLOWER]: Refusing update from remote peer 8e9d2970919a488da31c348bc8bf2d8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:45.959960 9052 raft_consensus.cc:1261] T 38791dcade5c442a8945100a6589624c P 2c2215bc64b14d40b64835326a0cebbd [term 1 FOLLOWER]: Refusing update from remote peer 8e9d2970919a488da31c348bc8bf2d8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:45.960515 9292 consensus_queue.cc:1035] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:57:45.960754 9269 consensus_queue.cc:1035] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:57:45.962493 9268 tablet_replica.cc:439] T 38791dcade5c442a8945100a6589624c P 7df4a7cb612e48848f68934bcc690309: TxnStatusTablet state changed. Reason: New leader 8e9d2970919a488da31c348bc8bf2d8e. Latest consensus state: current_term: 1 leader_uuid: "8e9d2970919a488da31c348bc8bf2d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } } }
I20260929 17:57:45.962643 9268 tablet_replica.cc:442] T 38791dcade5c442a8945100a6589624c P 7df4a7cb612e48848f68934bcc690309: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:57:45.962824 9267 tablet_replica.cc:439] T 38791dcade5c442a8945100a6589624c P 2c2215bc64b14d40b64835326a0cebbd: TxnStatusTablet state changed. Reason: New leader 8e9d2970919a488da31c348bc8bf2d8e. Latest consensus state: current_term: 1 leader_uuid: "8e9d2970919a488da31c348bc8bf2d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } } }
I20260929 17:57:45.962929 9267 tablet_replica.cc:442] T 38791dcade5c442a8945100a6589624c P 2c2215bc64b14d40b64835326a0cebbd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:57:45.966984 9269 tablet_replica.cc:439] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8e9d2970919a488da31c348bc8bf2d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } } }
I20260929 17:57:45.967125 9269 tablet_replica.cc:442] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e: This TxnStatusTablet replica's current role is: LEADER
I20260929 17:57:45.968628 9267 tablet_replica.cc:439] T 38791dcade5c442a8945100a6589624c P 2c2215bc64b14d40b64835326a0cebbd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8e9d2970919a488da31c348bc8bf2d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } } }
I20260929 17:57:45.968776 9267 tablet_replica.cc:442] T 38791dcade5c442a8945100a6589624c P 2c2215bc64b14d40b64835326a0cebbd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:57:45.975870 9292 tablet_replica.cc:439] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8e9d2970919a488da31c348bc8bf2d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } } }
I20260929 17:57:45.976027 9292 tablet_replica.cc:442] T 38791dcade5c442a8945100a6589624c P 8e9d2970919a488da31c348bc8bf2d8e: This TxnStatusTablet replica's current role is: LEADER
I20260929 17:57:45.976362 9308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:45.972554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57048 (local address 127.7.118.1:42559)
0929 17:57:45.972686 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:45.972692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:45.972750 (+ 58us) server_negotiation.cc:407] Connection header received
0929 17:57:45.972924 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:45.972927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:45.972987 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:45.973070 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:57:45.973323 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.973838 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.974910 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.975225 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.975635 (+ 410us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:45.975732 (+ 97us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:57:45.975837 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:57:45.975944 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:45.976152 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 17:57:45.976211 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 17:57:45.977497 9268 tablet_replica.cc:439] T 38791dcade5c442a8945100a6589624c P 7df4a7cb612e48848f68934bcc690309: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8e9d2970919a488da31c348bc8bf2d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9d2970919a488da31c348bc8bf2d8e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42559 } } peers { permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 } } peers { permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 } } }
I20260929 17:57:45.977629 9268 tablet_replica.cc:442] T 38791dcade5c442a8945100a6589624c P 7df4a7cb612e48848f68934bcc690309: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:57:45.990147 9308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:45.985951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:56861 (local address 127.7.118.1:42559)
0929 17:57:45.986081 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:45.986085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:45.986411 (+ 326us) server_negotiation.cc:407] Connection header received
0929 17:57:45.986515 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:45.986519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:45.986572 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:45.986668 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:57:45.986952 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.987747 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.988727 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.989082 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.989196 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:45.989538 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:57:45.989638 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:57:45.989722 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:45.989932 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 17:57:45.989993 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":53}
I20260929 17:57:45.990150 9314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:45.985911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:42559 (local address 127.7.118.1:56861)
0929 17:57:45.986221 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:45.986237 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:45.986435 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:45.986684 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:45.986686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:45.986692 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:57:45.986841 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:45.986846 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.987949 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.987952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:45.988589 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:45.988599 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:45.989238 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:45.989242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:45.989333 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:45.989387 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:57:45.989812 (+ 425us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:57:45.989819 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 17:57:45.989899 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 17:57:45.989981 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":25,"thread_start_us":128,"threads_started":1}
I20260929 17:57:45.998313 9192 raft_consensus.cc:1261] T 779d20f65f0441309692d77a92b36d88 P 7df4a7cb612e48848f68934bcc690309 [term 1 FOLLOWER]: Refusing update from remote peer 8e9d2970919a488da31c348bc8bf2d8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:45.998303 9052 raft_consensus.cc:1261] T 779d20f65f0441309692d77a92b36d88 P 2c2215bc64b14d40b64835326a0cebbd [term 1 FOLLOWER]: Refusing update from remote peer 8e9d2970919a488da31c348bc8bf2d8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:45.998768 9311 consensus_queue.cc:1035] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df4a7cb612e48848f68934bcc690309" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:57:45.998982 9315 consensus_queue.cc:1035] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c2215bc64b14d40b64835326a0cebbd" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 43843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:57:46.048059 8755 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:57:46.048276 8755 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:57:46.050827 8755 catalog_manager.cc:5782] T 00000000000000000000000000000000 P acb652fa8bad4fc99cc18238f6297523: Sending DeleteTablet for 3 replicas of tablet 779d20f65f0441309692d77a92b36d88
I20260929 17:57:46.051919 8891 tablet_service.cc:1514] Processing DeleteTablet for tablet 779d20f65f0441309692d77a92b36d88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:46 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:57044
I20260929 17:57:46.051914 9032 tablet_service.cc:1514] Processing DeleteTablet for tablet 779d20f65f0441309692d77a92b36d88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:46 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:46610
I20260929 17:57:46.052245 9172 tablet_service.cc:1514] Processing DeleteTablet for tablet 779d20f65f0441309692d77a92b36d88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:46 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:54764
I20260929 17:57:46.052286 9321 tablet_replica.cc:330] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e: stopping tablet replica
I20260929 17:57:46.052428 9321 raft_consensus.cc:2229] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:57:46.052664 9320 tablet_replica.cc:330] T 779d20f65f0441309692d77a92b36d88 P 2c2215bc64b14d40b64835326a0cebbd: stopping tablet replica
I20260929 17:57:46.052794 9320 raft_consensus.cc:2229] T 779d20f65f0441309692d77a92b36d88 P 2c2215bc64b14d40b64835326a0cebbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:57:46.052789 9321 raft_consensus.cc:2258] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:57:46.052920 9320 raft_consensus.cc:2258] T 779d20f65f0441309692d77a92b36d88 P 2c2215bc64b14d40b64835326a0cebbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:57:46.053630 9321 ts_tablet_manager.cc:1905] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:57:46.053673 9320 ts_tablet_manager.cc:1905] T 779d20f65f0441309692d77a92b36d88 P 2c2215bc64b14d40b64835326a0cebbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:57:46.057137 9320 ts_tablet_manager.cc:1918] T 779d20f65f0441309692d77a92b36d88 P 2c2215bc64b14d40b64835326a0cebbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:57:46.057255 9320 log.cc:1196] T 779d20f65f0441309692d77a92b36d88 P 2c2215bc64b14d40b64835326a0cebbd: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-1/wal/wals/779d20f65f0441309692d77a92b36d88
I20260929 17:57:46.057164 9321 ts_tablet_manager.cc:1918] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:57:46.057505 9321 log.cc:1196] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-0/wal/wals/779d20f65f0441309692d77a92b36d88
I20260929 17:57:46.057539 9320 ts_tablet_manager.cc:1939] T 779d20f65f0441309692d77a92b36d88 P 2c2215bc64b14d40b64835326a0cebbd: Deleting consensus metadata
I20260929 17:57:46.057888 9321 ts_tablet_manager.cc:1939] T 779d20f65f0441309692d77a92b36d88 P 8e9d2970919a488da31c348bc8bf2d8e: Deleting consensus metadata
I20260929 17:57:46.058914 8742 catalog_manager.cc:4841] TS 2c2215bc64b14d40b64835326a0cebbd (127.7.118.2:43843): tablet 779d20f65f0441309692d77a92b36d88 (table test-table [id=67f26a184a8843bb93ca2f32db942b5c]) successfully deleted
I20260929 17:57:46.059162 8742 catalog_manager.cc:4841] TS 8e9d2970919a488da31c348bc8bf2d8e (127.7.118.1:42559): tablet 779d20f65f0441309692d77a92b36d88 (table test-table [id=67f26a184a8843bb93ca2f32db942b5c]) successfully deleted
I20260929 17:57:46.052841 9323 tablet_replica.cc:330] T 779d20f65f0441309692d77a92b36d88 P 7df4a7cb612e48848f68934bcc690309: stopping tablet replica
I20260929 17:57:46.059643 9323 raft_consensus.cc:2229] T 779d20f65f0441309692d77a92b36d88 P 7df4a7cb612e48848f68934bcc690309 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:57:46.059811 9323 raft_consensus.cc:2258] T 779d20f65f0441309692d77a92b36d88 P 7df4a7cb612e48848f68934bcc690309 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:57:46.060735 9323 ts_tablet_manager.cc:1905] T 779d20f65f0441309692d77a92b36d88 P 7df4a7cb612e48848f68934bcc690309: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 17:57:46 dist-test-slave-r7ph krb5kdc[8710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704664, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.118.1@KRBTEST.COM
I20260929 17:57:46.064221 9323 ts_tablet_manager.cc:1918] T 779d20f65f0441309692d77a92b36d88 P 7df4a7cb612e48848f68934bcc690309: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:57:46.064327 9323 log.cc:1196] T 779d20f65f0441309692d77a92b36d88 P 7df4a7cb612e48848f68934bcc690309: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704657566865-7640-0/minicluster-data/ts-2/wal/wals/779d20f65f0441309692d77a92b36d88
I20260929 17:57:46.064606 9323 ts_tablet_manager.cc:1939] T 779d20f65f0441309692d77a92b36d88 P 7df4a7cb612e48848f68934bcc690309: Deleting consensus metadata
I20260929 17:57:46.065496 8739 catalog_manager.cc:4841] TS 7df4a7cb612e48848f68934bcc690309 (127.7.118.3:34861): tablet 779d20f65f0441309692d77a92b36d88 (table test-table [id=67f26a184a8843bb93ca2f32db942b5c]) successfully deleted
I20260929 17:57:46.066141 9308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:46.053278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57060 (local address 127.7.118.1:42559)
0929 17:57:46.053415 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:46.053420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:46.054038 (+ 618us) server_negotiation.cc:407] Connection header received
0929 17:57:46.054236 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:46.054242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:46.054326 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:46.054388 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:46.055372 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:46.056096 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:46.057302 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:46.057692 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:46.057782 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:46.061973 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:46.061984 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:46.062005 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:46.062041 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:46.063919 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:46.064480 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:46.064488 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:46.064490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:46.064601 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:46.064944 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:46.064948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:46.064950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:46.065235 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:46.065519 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:46.065824 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 17:57:46.065963 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 17:57:46.066641 8952 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57060
I20260929 17:57:46.077620 9300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:46.068637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57448 (local address 127.7.118.62:33793)
0929 17:57:46.068822 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:46.068827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:46.068850 (+ 23us) server_negotiation.cc:407] Connection header received
0929 17:57:46.069008 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:46.069012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:46.069069 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:46.069177 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:46.069945 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:46.070854 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:46.071805 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:46.072185 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:46.072333 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:46.073838 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:46.073843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:46.073846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:46.073894 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:46.075851 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:46.076348 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:46.076352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:46.076354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:46.076459 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:46.076730 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:46.076733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:46.076735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:46.076967 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:46.077098 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:46.077301 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 17:57:46.077445 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 17:57:46.078053 8755 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57448
I20260929 17:57:46.079475 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 8827
I20260929 17:57:46.086596 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 8967
I20260929 17:57:46.093443 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 9107
I20260929 17:57:46.099742 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 8723
2026-09-29T17:57:46Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3427 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:46 dist-test-slave-r7ph krb5kdc[9329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:46 dist-test-slave-r7ph krb5kdc[9329](info): set up 2 sockets
Sep 29 17:57:46 dist-test-slave-r7ph krb5kdc[9329](info): commencing operation
krb5kdc: starting...
W20260929 17:57:47.178212 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 17:57:47 dist-test-slave-r7ph krb5kdc[9329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 17:57:47.315941 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:46455
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:46455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:47.437424 9342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:47.437798 9342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:47.437891 9342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:47.442196 9342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:47.442374 9342 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 17:57:47.442411 9342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:47.442471 9342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:47.442508 9342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:47.442550 9342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:47.448351 9342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:46455
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:46455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:47.449571 9342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:47.450747 9342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:47.459935 9347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:47.459935 9348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:47.459939 9350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:47.460181 9342 server_base.cc:1034] running on GCE node
I20260929 17:57:47.460932 9342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:47.462232 9342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:47.463478 9342 hybrid_clock.cc:648] HybridClock initialized: now 1790704667463441 us; error 62 us; skew 500 ppm
Sep 29 17:57:47 dist-test-slave-r7ph krb5kdc[9329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704667, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:47.466774 9342 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 17:57:47.467792 9342 webserver.cc:460] Webserver started at http://127.7.118.62:45403/ using document root <none> and password file <none>
I20260929 17:57:47.468422 9342 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:47.468518 9342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:47.468757 9342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:47.471181 9342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "c7008ea9982e4f7eaa7ef48b63036862"
format_stamp: "Formatted at 2026-09-29 17:57:47 on dist-test-slave-r7ph"
server_key: "da1b903e792029b07a6f4c70376f0ba4"
server_key_iv: "c4bbc1fff896bc5ed66b1e1f0505a686"
server_key_version: "encryptionkey@0"
I20260929 17:57:47.471809 9342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "c7008ea9982e4f7eaa7ef48b63036862"
format_stamp: "Formatted at 2026-09-29 17:57:47 on dist-test-slave-r7ph"
server_key: "da1b903e792029b07a6f4c70376f0ba4"
server_key_iv: "c4bbc1fff896bc5ed66b1e1f0505a686"
server_key_version: "encryptionkey@0"
I20260929 17:57:47.475986 9342 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:57:47.478771 9357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:47.480212 9342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:57:47.480398 9342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "c7008ea9982e4f7eaa7ef48b63036862"
format_stamp: "Formatted at 2026-09-29 17:57:47 on dist-test-slave-r7ph"
server_key: "da1b903e792029b07a6f4c70376f0ba4"
server_key_iv: "c4bbc1fff896bc5ed66b1e1f0505a686"
server_key_version: "encryptionkey@0"
I20260929 17:57:47.480544 9342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:47.506114 9342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:47.509632 9342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:47.509907 9342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:47.518435 9342 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:46455
I20260929 17:57:47.518446 9419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:46455 every 8 connection(s)
I20260929 17:57:47.519491 9342 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:47.522763 9420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:47.524027 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 9342
I20260929 17:57:47.524183 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 17:57:47.524461 7640 external_mini_cluster.cc:1428] Setting key f031ba14530a039a5045665a1d45218e
I20260929 17:57:47.530058 9420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862: Bootstrap starting.
I20260929 17:57:47.532923 9420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:47.533978 9420 log.cc:824] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862: Log is configured to *not* fsync() on all Append() calls
Sep 29 17:57:47 dist-test-slave-r7ph krb5kdc[9329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:57:47.536695 9420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862: No bootstrap required, opened a new log
I20260929 17:57:47.541101 9420 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7008ea9982e4f7eaa7ef48b63036862" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 46455 } }
I20260929 17:57:47.541404 9420 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:47.541491 9420 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7008ea9982e4f7eaa7ef48b63036862, State: Initialized, Role: FOLLOWER
I20260929 17:57:47.542023 9420 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7008ea9982e4f7eaa7ef48b63036862" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 46455 } }
I20260929 17:57:47.542192 9420 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:47.542191 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:47.525983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40034 (local address 127.7.118.62:46455)
0929 17:57:47.526467 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:47.526488 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:47.526533 (+ 45us) server_negotiation.cc:407] Connection header received
0929 17:57:47.527395 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:47.527421 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:47.527871 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:47.528289 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:47.529034 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:47.530554 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:47.531560 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:47.532273 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:47.532448 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:47.535647 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:47.535654 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:47.535672 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:47.535709 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:47.538537 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:47.539117 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:47.539123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:47.539130 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:47.539289 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:47.539654 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:47.539658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:47.539660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:47.540215 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:47.540419 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:47.540733 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 17:57:47.541081 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":114,"threads_started":1}
I20260929 17:57:47.542316 9420 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:47.542405 9420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:47.543409 9420 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7008ea9982e4f7eaa7ef48b63036862" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 46455 } }
I20260929 17:57:47.543797 9420 leader_election.cc:302] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7008ea9982e4f7eaa7ef48b63036862; no voters:
I20260929 17:57:47.544160 9420 leader_election.cc:288] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:47.544309 9425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:47.544572 9425 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862 [term 1 LEADER]: Becoming Leader. State: Replica: c7008ea9982e4f7eaa7ef48b63036862, State: Running, Role: LEADER
I20260929 17:57:47.545022 9425 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7008ea9982e4f7eaa7ef48b63036862" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 46455 } }
I20260929 17:57:47.545493 9420 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:47.546993 9426 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7008ea9982e4f7eaa7ef48b63036862. Latest consensus state: current_term: 1 leader_uuid: "c7008ea9982e4f7eaa7ef48b63036862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7008ea9982e4f7eaa7ef48b63036862" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 46455 } } }
I20260929 17:57:47.547044 9427 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7008ea9982e4f7eaa7ef48b63036862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7008ea9982e4f7eaa7ef48b63036862" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 46455 } } }
I20260929 17:57:47.547140 9426 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:47.547148 9427 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862 [sys.catalog]: This master's current role is: LEADER
W20260929 17:57:47.550529 9440 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:57:47.550637 9440 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:57:47.550726 9437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:47.551440 9437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:47.557502 9437 catalog_manager.cc:1347] Generated new cluster ID: 9ec604b4961e435b95ca376c365fce71
I20260929 17:57:47.557607 9437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:47.585542 9437 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:47.586650 9437 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:47.594811 9437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862: Generated new TSK 0
I20260929 17:57:47.596335 9437 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.118.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
I20260929 17:57:47.634419 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:0
--local_ip_for_outbound_sockets=127.7.118.1
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:46455
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:47.759006 9446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:47.759341 9446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:47.759393 9446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:47.764411 9446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:47.764518 9446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:47.764566 9446 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 17:57:47.764582 9446 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 17:57:47.764617 9446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 17:57:47.770603 9446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.118.62:46455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:47.771739 9446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:47.772795 9446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:47.783491 9454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:47.783424 9451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:47.783450 9452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:47.783993 9446 server_base.cc:1034] running on GCE node
I20260929 17:57:47.784518 9446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:47.785395 9446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:47.786572 9446 hybrid_clock.cc:648] HybridClock initialized: now 1790704667786557 us; error 33 us; skew 500 ppm
Sep 29 17:57:47 dist-test-slave-r7ph krb5kdc[9329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704667, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:47.789966 9446 init.cc:377] Logged in from keytab as kudu/127.7.118.1@KRBTEST.COM (short username kudu)
I20260929 17:57:47.791085 9446 webserver.cc:460] Webserver started at http://127.7.118.1:45799/ using document root <none> and password file <none>
I20260929 17:57:47.791738 9446 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:47.791842 9446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:47.792083 9446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:47.794498 9446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-0/data/instance:
uuid: "0aaa166ca9684c14a9cf42334976693a"
format_stamp: "Formatted at 2026-09-29 17:57:47 on dist-test-slave-r7ph"
server_key: "5ca390c53b624f32d4934259dc026636"
server_key_iv: "4541e30ae9ade15cbf8b429a168ee2e9"
server_key_version: "encryptionkey@0"
I20260929 17:57:47.795178 9446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance:
uuid: "0aaa166ca9684c14a9cf42334976693a"
format_stamp: "Formatted at 2026-09-29 17:57:47 on dist-test-slave-r7ph"
server_key: "5ca390c53b624f32d4934259dc026636"
server_key_iv: "4541e30ae9ade15cbf8b429a168ee2e9"
server_key_version: "encryptionkey@0"
I20260929 17:57:47.799944 9446 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:57:47.803381 9461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:47.805269 9446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 17:57:47.805466 9446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "0aaa166ca9684c14a9cf42334976693a"
format_stamp: "Formatted at 2026-09-29 17:57:47 on dist-test-slave-r7ph"
server_key: "5ca390c53b624f32d4934259dc026636"
server_key_iv: "4541e30ae9ade15cbf8b429a168ee2e9"
server_key_version: "encryptionkey@0"
I20260929 17:57:47.805619 9446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:47.833814 9446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:47.838045 9446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:47.838373 9446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:47.840173 9446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:57:47.840265 9446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:47.840343 9446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:57:47.840394 9446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 17:57:47 dist-test-slave-r7ph krb5kdc[9329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704667, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:57:47.853543 9446 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:44893
I20260929 17:57:47.853888 9576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:44893 every 8 connection(s)
I20260929 17:57:47.854956 9446 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 17:57:47.857270 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:47.841162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:50403 (local address 127.7.118.62:46455)
0929 17:57:47.841344 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:47.841348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:47.842684 (+ 1336us) server_negotiation.cc:407] Connection header received
0929 17:57:47.843935 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:47.843939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:47.844023 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:47.844090 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:47.845988 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:47.846906 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:47.848444 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:47.848728 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:47.848837 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:47.852567 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:47.852570 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:47.852573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:47.852602 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:47.854676 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:47.855353 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:47.855361 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:47.855372 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:47.855505 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:47.855995 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:47.856000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:47.856002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:47.856231 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:47.856354 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:47.856926 (+ 572us) server_negotiation.cc:299] Negotiation successful
0929 17:57:47.857080 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 17:57:47.858165 9470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:47.841846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:46455 (local address 127.7.118.1:50403)
0929 17:57:47.842511 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:47.842561 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:47.843593 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:47.844505 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:47.844517 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:47.845061 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:47.845796 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:47.845816 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:47.847057 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:47.847062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:47.848242 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:47.848254 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:47.848894 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:47.848898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:47.849129 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:47.849928 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:47.849952 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:47.852284 (+ 2332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:47.854842 (+ 2558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:47.854848 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:47.854862 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:47.855224 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:47.855637 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:47.855641 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:47.855643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:47.855863 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:47.856352 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:47.856360 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:47.856648 (+ 288us) client_negotiation.cc:769] Sending connection context
0929 17:57:47.856869 (+ 221us) client_negotiation.cc:240] Negotiation successful
0929 17:57:47.857245 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":163,"threads_started":1}
I20260929 17:57:47.859798 9577 heartbeater.cc:347] Connected to a master server at 127.7.118.62:46455
I20260929 17:57:47.859973 9577 heartbeater.cc:464] Registering TS with master...
I20260929 17:57:47.860651 9577 heartbeater.cc:511] Master 127.7.118.62:46455 requested a full tablet report, sending...
I20260929 17:57:47.862838 9373 ts_manager.cc:194] Registered new tserver with Master: 0aaa166ca9684c14a9cf42334976693a (127.7.118.1:44893)
I20260929 17:57:47.863649 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 9446
I20260929 17:57:47.863794 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance
I20260929 17:57:47.864105 7640 external_mini_cluster.cc:1428] Setting key 7689baef11486518feb96873f6284c1c
I20260929 17:57:47.864516 9373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.1@KRBTEST.COM'} at 127.7.118.1:50403
I20260929 17:57:47.872776 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:47.863211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:46455 (local address 127.7.118.1:48653)
0929 17:57:47.863550 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:47.863566 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:47.863737 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:47.864091 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:47.864094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:47.864322 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:47.864455 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:47.864463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:47.865501 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:47.865507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:47.866586 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:47.866603 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:47.867227 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:47.867234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:47.867386 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:47.868047 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:47.868060 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:47.868546 (+ 486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:47.870770 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:47.870774 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:47.870776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:47.871011 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:47.871511 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:47.871514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:47.871516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:47.871611 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:47.872233 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:47.872235 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:47.872335 (+ 100us) client_negotiation.cc:769] Sending connection context
0929 17:57:47.872427 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 17:57:47.872569 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":150,"threads_started":1}
I20260929 17:57:47.872929 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:47.863157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:48653 (local address 127.7.118.62:46455)
0929 17:57:47.863296 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:47.863304 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:47.863646 (+ 342us) server_negotiation.cc:407] Connection header received
0929 17:57:47.863854 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:47.863859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:47.863906 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:47.863957 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:47.864581 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:47.865307 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:47.866750 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:47.867074 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:47.867190 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:47.868707 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:47.868714 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:47.868728 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:47.868764 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:47.870619 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:47.871179 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:47.871196 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:47.871200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:47.871326 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:47.871736 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:47.871744 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:47.871748 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:47.872063 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:47.872229 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:47.872534 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 17:57:47.872708 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
WARNING: no policy specified for kudu/127.7.118.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
I20260929 17:57:47.898574 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:0
--local_ip_for_outbound_sockets=127.7.118.2
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:46455
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:48.025280 9586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:48.025650 9586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:48.025715 9586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:48.031504 9586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:48.031605 9586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:48.031656 9586 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 17:57:48.031673 9586 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 17:57:48.031781 9586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 17:57:48.039216 9586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.118.62:46455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:48.040436 9586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:48.041983 9586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:48.053615 9592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:48.054105 9591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:48.054312 9586 server_base.cc:1034] running on GCE node
W20260929 17:57:48.054316 9594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:48.054872 9586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:48.055660 9586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:48.057054 9586 hybrid_clock.cc:648] HybridClock initialized: now 1790704668057034 us; error 33 us; skew 500 ppm
Sep 29 17:57:48 dist-test-slave-r7ph krb5kdc[9329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704668, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:48.060699 9586 init.cc:377] Logged in from keytab as kudu/127.7.118.2@KRBTEST.COM (short username kudu)
I20260929 17:57:48.061928 9586 webserver.cc:460] Webserver started at http://127.7.118.2:43671/ using document root <none> and password file <none>
I20260929 17:57:48.062678 9586 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:48.062812 9586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:48.063076 9586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:48.065543 9586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-1/data/instance:
uuid: "965153f8a69c4e6aa33715bb5a033801"
format_stamp: "Formatted at 2026-09-29 17:57:48 on dist-test-slave-r7ph"
server_key: "1242ce4e64494a428c78364f888b49bb"
server_key_iv: "d574a91a67c95f1e32df7e8b2290dd70"
server_key_version: "encryptionkey@0"
I20260929 17:57:48.066211 9586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance:
uuid: "965153f8a69c4e6aa33715bb5a033801"
format_stamp: "Formatted at 2026-09-29 17:57:48 on dist-test-slave-r7ph"
server_key: "1242ce4e64494a428c78364f888b49bb"
server_key_iv: "d574a91a67c95f1e32df7e8b2290dd70"
server_key_version: "encryptionkey@0"
I20260929 17:57:48.071244 9586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 17:57:48.074415 9601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:48.076061 9586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 17:57:48.076260 9586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "965153f8a69c4e6aa33715bb5a033801"
format_stamp: "Formatted at 2026-09-29 17:57:48 on dist-test-slave-r7ph"
server_key: "1242ce4e64494a428c78364f888b49bb"
server_key_iv: "d574a91a67c95f1e32df7e8b2290dd70"
server_key_version: "encryptionkey@0"
I20260929 17:57:48.076417 9586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:48.089783 9586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:48.094202 9586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:48.094463 9586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:48.096145 9586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:57:48.096232 9586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:48.096295 9586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:57:48.096345 9586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 17:57:48 dist-test-slave-r7ph krb5kdc[9329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704668, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:57:48.107777 9586 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:42821
I20260929 17:57:48.107856 9716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:42821 every 8 connection(s)
I20260929 17:57:48.109081 9586 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 17:57:48.111799 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.097247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:42611 (local address 127.7.118.62:46455)
0929 17:57:48.097446 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:48.097451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:48.098408 (+ 957us) server_negotiation.cc:407] Connection header received
0929 17:57:48.099563 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:48.099567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:48.099651 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:48.099762 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:48.101290 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.102202 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.103935 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.104180 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.104311 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.107227 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:48.107231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:48.107233 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:48.107269 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:48.109153 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:48.109934 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:48.109938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:48.109940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:48.110043 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:48.110537 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:48.110540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:48.110541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:48.110745 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:48.110822 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:48.111487 (+ 665us) server_negotiation.cc:299] Negotiation successful
0929 17:57:48.111617 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 17:57:48.112756 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.097705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:46455 (local address 127.7.118.2:42611)
0929 17:57:48.098194 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:48.098281 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:48.099287 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:48.100013 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:48.100024 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:48.100529 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:48.101126 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.101141 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.102405 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.102409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.103781 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.103794 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.104369 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.104373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.104566 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.105252 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:48.105269 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:48.107010 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:48.109379 (+ 2369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:48.109385 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:48.109403 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:48.109828 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:48.110153 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:48.110159 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:48.110163 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:48.110406 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:48.110886 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:48.110893 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:48.111204 (+ 311us) client_negotiation.cc:769] Sending connection context
0929 17:57:48.111401 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 17:57:48.111760 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":139,"threads_started":1}
I20260929 17:57:48.114509 9717 heartbeater.cc:347] Connected to a master server at 127.7.118.62:46455
I20260929 17:57:48.114725 9717 heartbeater.cc:464] Registering TS with master...
I20260929 17:57:48.115409 9717 heartbeater.cc:511] Master 127.7.118.62:46455 requested a full tablet report, sending...
I20260929 17:57:48.116676 9374 ts_manager.cc:194] Registered new tserver with Master: 965153f8a69c4e6aa33715bb5a033801 (127.7.118.2:42821)
I20260929 17:57:48.116816 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 9586
I20260929 17:57:48.116920 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance
I20260929 17:57:48.117244 7640 external_mini_cluster.cc:1428] Setting key 3868e4644e636068a6521c65a2a16391
I20260929 17:57:48.117524 9374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.2@KRBTEST.COM'} at 127.7.118.2:42611
I20260929 17:57:48.126226 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.116961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:50467 (local address 127.7.118.62:46455)
0929 17:57:48.117075 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:48.117079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:48.117479 (+ 400us) server_negotiation.cc:407] Connection header received
0929 17:57:48.117590 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:48.117596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:48.117650 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:48.117771 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:48.118342 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.119032 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.120384 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.120630 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.120762 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.122243 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:48.122315 (+ 72us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:48.122319 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:48.122356 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:48.124113 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:48.124735 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:48.124739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:48.124740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:48.124848 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:48.125216 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:48.125219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:48.125220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:48.125486 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:48.125580 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:48.125984 (+ 404us) server_negotiation.cc:299] Negotiation successful
0929 17:57:48.126098 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:57:48.126389 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.116888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:46455 (local address 127.7.118.2:50467)
0929 17:57:48.117378 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:48.117393 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:48.117503 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:48.117793 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:48.117796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:48.118011 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:48.118164 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.118173 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.119211 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.119215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.120232 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.120246 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.120793 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.120800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.120918 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.121486 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:48.121508 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:48.122021 (+ 513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:48.124250 (+ 2229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:48.124261 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:48.124266 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:48.124582 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:48.124959 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:48.124965 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:48.124969 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:48.125086 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:48.125598 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:48.125601 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:48.125740 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 17:57:48.125883 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 17:57:48.126087 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":135,"threads_started":1}
WARNING: no policy specified for kudu/127.7.118.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
I20260929 17:57:48.153990 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:0
--local_ip_for_outbound_sockets=127.7.118.3
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:46455
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:48.285324 9726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:48.285645 9726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:48.285698 9726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:48.290154 9726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:48.290436 9726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:48.290566 9726 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 17:57:48.290587 9726 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 17:57:48.290625 9726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 17:57:48.299023 9726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.118.62:46455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:48.300180 9726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:48.301433 9726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:48.313052 9731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:48.313045 9732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:48.313346 9726 server_base.cc:1034] running on GCE node
W20260929 17:57:48.313895 9734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:48.314558 9726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:48.315435 9726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:48.316664 9726 hybrid_clock.cc:648] HybridClock initialized: now 1790704668316640 us; error 39 us; skew 500 ppm
Sep 29 17:57:48 dist-test-slave-r7ph krb5kdc[9329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704668, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:48.320001 9726 init.cc:377] Logged in from keytab as kudu/127.7.118.3@KRBTEST.COM (short username kudu)
I20260929 17:57:48.321182 9726 webserver.cc:460] Webserver started at http://127.7.118.3:39497/ using document root <none> and password file <none>
I20260929 17:57:48.321822 9726 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:48.321925 9726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:48.322173 9726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:48.324544 9726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-2/data/instance:
uuid: "9e71099a7b5c462db122ac2a7e033be9"
format_stamp: "Formatted at 2026-09-29 17:57:48 on dist-test-slave-r7ph"
server_key: "af58633def0177753a04bcd0807202b4"
server_key_iv: "ff8a23839b5bfb735367b64b2e89ca8f"
server_key_version: "encryptionkey@0"
I20260929 17:57:48.325165 9726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance:
uuid: "9e71099a7b5c462db122ac2a7e033be9"
format_stamp: "Formatted at 2026-09-29 17:57:48 on dist-test-slave-r7ph"
server_key: "af58633def0177753a04bcd0807202b4"
server_key_iv: "ff8a23839b5bfb735367b64b2e89ca8f"
server_key_version: "encryptionkey@0"
I20260929 17:57:48.329295 9726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 17:57:48.332638 9741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:48.334542 9726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:57:48.334746 9726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "9e71099a7b5c462db122ac2a7e033be9"
format_stamp: "Formatted at 2026-09-29 17:57:48 on dist-test-slave-r7ph"
server_key: "af58633def0177753a04bcd0807202b4"
server_key_iv: "ff8a23839b5bfb735367b64b2e89ca8f"
server_key_version: "encryptionkey@0"
I20260929 17:57:48.334918 9726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:48.359644 9726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:48.363575 9726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:48.363811 9726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:48.365625 9726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:57:48.365720 9726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:48.365801 9726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:57:48.365845 9726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 17:57:48 dist-test-slave-r7ph krb5kdc[9329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704668, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:57:48.378521 9726 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:45999
I20260929 17:57:48.378571 9856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:45999 every 8 connection(s)
I20260929 17:57:48.379865 9726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 17:57:48.381578 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.366570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:39935 (local address 127.7.118.62:46455)
0929 17:57:48.366739 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:48.366744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:48.367951 (+ 1207us) server_negotiation.cc:407] Connection header received
0929 17:57:48.369029 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:48.369036 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:48.369103 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:48.369235 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:48.370839 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.371450 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.372845 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.373137 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.373276 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.376491 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:48.376495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:48.376498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:48.376537 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:48.378707 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:48.379301 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:48.379306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:48.379308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:48.379426 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:48.379871 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:48.379877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:48.379882 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:48.380165 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:48.380301 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:48.381208 (+ 907us) server_negotiation.cc:299] Negotiation successful
0929 17:57:48.381383 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 17:57:48.382426 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 9726
I20260929 17:57:48.382555 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance
I20260929 17:57:48.382835 9750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.367209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:46455 (local address 127.7.118.3:39935)
0929 17:57:48.367767 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:48.367819 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:48.368759 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:48.369681 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:48.369688 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:48.370048 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:48.370645 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.370684 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.371586 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.371591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.372710 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.372721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.373331 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.373335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.373572 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.374542 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:48.374561 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:48.376257 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:48.378850 (+ 2593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:48.378856 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:48.378869 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:48.379170 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:48.379568 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:48.379570 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:48.379572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:48.379746 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:48.380385 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:48.380402 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:48.380794 (+ 392us) client_negotiation.cc:769] Sending connection context
0929 17:57:48.381134 (+ 340us) client_negotiation.cc:240] Negotiation successful
0929 17:57:48.381525 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":157,"threads_started":1}
I20260929 17:57:48.382894 7640 external_mini_cluster.cc:1428] Setting key 85724917c52b5d5f102e96faaa58289e
I20260929 17:57:48.384433 9857 heartbeater.cc:347] Connected to a master server at 127.7.118.62:46455
I20260929 17:57:48.384698 9857 heartbeater.cc:464] Registering TS with master...
I20260929 17:57:48.385324 9857 heartbeater.cc:511] Master 127.7.118.62:46455 requested a full tablet report, sending...
I20260929 17:57:48.386744 9373 ts_manager.cc:194] Registered new tserver with Master: 9e71099a7b5c462db122ac2a7e033be9 (127.7.118.3:45999)
I20260929 17:57:48.387519 9373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.3@KRBTEST.COM'} at 127.7.118.3:39935
I20260929 17:57:48.396682 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.387454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:60195 (local address 127.7.118.62:46455)
0929 17:57:48.387598 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:48.387602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:48.387875 (+ 273us) server_negotiation.cc:407] Connection header received
0929 17:57:48.388047 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:48.388051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:48.388097 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:48.388199 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:48.388710 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.389296 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.390315 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.390601 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.390739 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.392309 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:48.392313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:48.392316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:48.392359 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:48.394548 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:48.395129 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:48.395133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:48.395136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:48.395226 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:48.395546 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:48.395549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:48.395551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:48.395828 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:48.395941 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:48.396336 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 17:57:48.396472 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 17:57:48.396704 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.387326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:46455 (local address 127.7.118.3:60195)
0929 17:57:48.387768 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:48.387781 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:48.387929 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:48.388217 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:48.388220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:48.388472 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:48.388581 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.388587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.389458 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.389462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.390137 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.390149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.390749 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.390753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.390881 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.391552 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:48.391567 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:48.392094 (+ 527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:48.394722 (+ 2628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:48.394726 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:48.394729 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:48.395006 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:48.395325 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:48.395329 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:48.395331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:48.395431 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:48.395997 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:48.396000 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:48.396127 (+ 127us) client_negotiation.cc:769] Sending connection context
0929 17:57:48.396254 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 17:57:48.396466 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":124,"threads_started":1}
I20260929 17:57:48.397832 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 17:57:48 dist-test-slave-r7ph krb5kdc[9329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 17:57:48 dist-test-slave-r7ph krb5kdc[9329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:57:48.429121 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.418171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40042 (local address 127.7.118.62:46455)
0929 17:57:48.418388 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:48.418392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:48.418408 (+ 16us) server_negotiation.cc:407] Connection header received
0929 17:57:48.418452 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:48.418460 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:48.418510 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:48.418597 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:48.419495 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.420087 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.421473 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.421834 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.421955 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.425754 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:48.425757 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:48.425759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:48.425795 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:48.427374 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:48.427868 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:48.427871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:48.427872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:48.427964 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:48.428263 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:48.428266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:48.428267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:48.428460 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:48.428576 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:48.428884 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 17:57:48.428994 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260929 17:57:48.433017 9373 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:57:48.436545 9373 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:57:48.453127 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.446803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42154 (local address 127.7.118.2:42821)
0929 17:57:48.447542 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:48.447555 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:48.447584 (+ 29us) server_negotiation.cc:407] Connection header received
0929 17:57:48.447777 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:48.447784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:48.448034 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:48.448190 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:48.448730 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.449765 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.451395 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.452312 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.452448 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.452631 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:48.452859 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 17:57:48.452964 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":99,"threads_started":1}
I20260929 17:57:48.453114 9875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.446605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:42821 (local address 127.0.0.1:42154)
0929 17:57:48.447346 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:48.447404 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:48.447550 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:48.448382 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:48.448391 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:48.448417 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:48.448589 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.448597 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.449878 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.449883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.451253 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.451264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.452458 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.452460 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.452564 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.452633 (+ 69us) client_negotiation.cc:769] Sending connection context
0929 17:57:48.452810 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 17:57:48.452956 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":73,"threads_started":1}
I20260929 17:57:48.453682 9874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.446697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:45999 (local address 127.0.0.1:41716)
0929 17:57:48.447620 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:48.447636 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:48.447709 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:48.448456 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:48.448459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:48.448567 (+ 108us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:48.448713 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.448719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.450301 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.450305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.451634 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.451690 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.453075 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.453079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.453363 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.453391 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 17:57:48.453467 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 17:57:48.453540 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":30,"thread_start_us":130,"threads_started":1}
I20260929 17:57:48.454209 9877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.447101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41716 (local address 127.7.118.3:45999)
0929 17:57:48.447796 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:48.447809 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:48.447830 (+ 21us) server_negotiation.cc:407] Connection header received
0929 17:57:48.447909 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:48.447913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:48.448106 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:48.448270 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:48.448926 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.450143 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.451852 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.452723 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.452848 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.453011 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:48.453953 (+ 942us) server_negotiation.cc:299] Negotiation successful
0929 17:57:48.454063 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":109,"threads_started":1}
I20260929 17:57:48.456106 9878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.447399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:44893 (local address 127.0.0.1:39880)
0929 17:57:48.448838 (+ 1439us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:48.448850 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:48.448940 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:48.449636 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:48.449640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:48.449659 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:48.449759 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.449765 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.451112 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.451116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.453798 (+ 2682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.453810 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.455625 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.455629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.455733 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.455757 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 17:57:48.455868 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 17:57:48.455955 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1380,"mutex_wait_us":1352,"thread_start_us":81,"threads_started":1}
I20260929 17:57:48.456373 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.447970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39880 (local address 127.7.118.1:44893)
0929 17:57:48.448558 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:48.448564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:48.448950 (+ 386us) server_negotiation.cc:407] Connection header received
0929 17:57:48.449044 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:48.449050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:48.449273 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:48.449480 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:48.449892 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.450973 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.453956 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.455479 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.455618 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.455773 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:48.456059 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 17:57:48.456191 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":93,"threads_started":1}
I20260929 17:57:48.456673 9648 tablet_service.cc:1467] Processing CreateTablet for tablet df719946884e4885b11f9a2cfbeeb8b7 (DEFAULT_TABLE table=test-table [id=ecfaa6977e40477aba1c4fd51a74b796]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:48.456882 9791 tablet_service.cc:1467] Processing CreateTablet for tablet df719946884e4885b11f9a2cfbeeb8b7 (DEFAULT_TABLE table=test-table [id=ecfaa6977e40477aba1c4fd51a74b796]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:48.457859 9791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df719946884e4885b11f9a2cfbeeb8b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:48.457854 9648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df719946884e4885b11f9a2cfbeeb8b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:48.459187 9511 tablet_service.cc:1467] Processing CreateTablet for tablet df719946884e4885b11f9a2cfbeeb8b7 (DEFAULT_TABLE table=test-table [id=ecfaa6977e40477aba1c4fd51a74b796]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:48.460270 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df719946884e4885b11f9a2cfbeeb8b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:48.466616 9881 tablet_bootstrap.cc:492] T df719946884e4885b11f9a2cfbeeb8b7 P 9e71099a7b5c462db122ac2a7e033be9: Bootstrap starting.
I20260929 17:57:48.466804 9880 tablet_bootstrap.cc:492] T df719946884e4885b11f9a2cfbeeb8b7 P 965153f8a69c4e6aa33715bb5a033801: Bootstrap starting.
I20260929 17:57:48.467700 9882 tablet_bootstrap.cc:492] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a: Bootstrap starting.
I20260929 17:57:48.469496 9880 tablet_bootstrap.cc:654] T df719946884e4885b11f9a2cfbeeb8b7 P 965153f8a69c4e6aa33715bb5a033801: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:48.469724 9882 tablet_bootstrap.cc:654] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:48.470374 9881 tablet_bootstrap.cc:654] T df719946884e4885b11f9a2cfbeeb8b7 P 9e71099a7b5c462db122ac2a7e033be9: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:48.470515 9882 log.cc:824] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:48.470515 9880 log.cc:824] T df719946884e4885b11f9a2cfbeeb8b7 P 965153f8a69c4e6aa33715bb5a033801: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:48.471254 9881 log.cc:824] T df719946884e4885b11f9a2cfbeeb8b7 P 9e71099a7b5c462db122ac2a7e033be9: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:48.472663 9880 tablet_bootstrap.cc:492] T df719946884e4885b11f9a2cfbeeb8b7 P 965153f8a69c4e6aa33715bb5a033801: No bootstrap required, opened a new log
I20260929 17:57:48.472659 9882 tablet_bootstrap.cc:492] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a: No bootstrap required, opened a new log
I20260929 17:57:48.472875 9880 ts_tablet_manager.cc:1397] T df719946884e4885b11f9a2cfbeeb8b7 P 965153f8a69c4e6aa33715bb5a033801: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 17:57:48.472875 9882 ts_tablet_manager.cc:1397] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 17:57:48.473294 9881 tablet_bootstrap.cc:492] T df719946884e4885b11f9a2cfbeeb8b7 P 9e71099a7b5c462db122ac2a7e033be9: No bootstrap required, opened a new log
I20260929 17:57:48.473519 9881 ts_tablet_manager.cc:1397] T df719946884e4885b11f9a2cfbeeb8b7 P 9e71099a7b5c462db122ac2a7e033be9: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260929 17:57:48.477003 9881 raft_consensus.cc:348] T df719946884e4885b11f9a2cfbeeb8b7 P 9e71099a7b5c462db122ac2a7e033be9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } } peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } }
I20260929 17:57:48.477337 9881 raft_consensus.cc:374] T df719946884e4885b11f9a2cfbeeb8b7 P 9e71099a7b5c462db122ac2a7e033be9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:48.477401 9881 raft_consensus.cc:729] T df719946884e4885b11f9a2cfbeeb8b7 P 9e71099a7b5c462db122ac2a7e033be9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e71099a7b5c462db122ac2a7e033be9, State: Initialized, Role: FOLLOWER
I20260929 17:57:48.477246 9880 raft_consensus.cc:348] T df719946884e4885b11f9a2cfbeeb8b7 P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } } peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } }
I20260929 17:57:48.477542 9880 raft_consensus.cc:374] T df719946884e4885b11f9a2cfbeeb8b7 P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:48.477599 9880 raft_consensus.cc:729] T df719946884e4885b11f9a2cfbeeb8b7 P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 965153f8a69c4e6aa33715bb5a033801, State: Initialized, Role: FOLLOWER
I20260929 17:57:48.477485 9882 raft_consensus.cc:348] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } } peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } }
I20260929 17:57:48.477737 9882 raft_consensus.cc:374] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:48.477790 9882 raft_consensus.cc:729] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aaa166ca9684c14a9cf42334976693a, State: Initialized, Role: FOLLOWER
I20260929 17:57:48.478020 9881 consensus_queue.cc:260] T df719946884e4885b11f9a2cfbeeb8b7 P 9e71099a7b5c462db122ac2a7e033be9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } } peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } }
I20260929 17:57:48.478049 9880 consensus_queue.cc:260] T df719946884e4885b11f9a2cfbeeb8b7 P 965153f8a69c4e6aa33715bb5a033801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } } peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } }
I20260929 17:57:48.478165 9882 consensus_queue.cc:260] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } } peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } }
I20260929 17:57:48.478889 9717 heartbeater.cc:503] Master 127.7.118.62:46455 was elected leader, sending a full tablet report...
I20260929 17:57:48.479230 9881 ts_tablet_manager.cc:1428] T df719946884e4885b11f9a2cfbeeb8b7 P 9e71099a7b5c462db122ac2a7e033be9: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260929 17:57:48.479238 9882 ts_tablet_manager.cc:1428] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 17:57:48.479230 9880 ts_tablet_manager.cc:1428] T df719946884e4885b11f9a2cfbeeb8b7 P 965153f8a69c4e6aa33715bb5a033801: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 17:57:48.479576 9577 heartbeater.cc:503] Master 127.7.118.62:46455 was elected leader, sending a full tablet report...
I20260929 17:57:48.479574 9857 heartbeater.cc:503] Master 127.7.118.62:46455 was elected leader, sending a full tablet report...
I20260929 17:57:48.486469 9887 raft_consensus.cc:482] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:57:48.486732 9887 raft_consensus.cc:504] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } } peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } }
I20260929 17:57:48.487983 9887 leader_election.cc:288] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 965153f8a69c4e6aa33715bb5a033801 (127.7.118.2:42821), 9e71099a7b5c462db122ac2a7e033be9 (127.7.118.3:45999)
I20260929 17:57:48.492350 9877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.488359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:46471 (local address 127.7.118.3:45999)
0929 17:57:48.488501 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:48.488506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:48.488811 (+ 305us) server_negotiation.cc:407] Connection header received
0929 17:57:48.488925 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:48.488928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:48.488980 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:48.489065 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:48.489565 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.490231 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.491373 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.491788 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.491899 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.491963 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:48.492136 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 17:57:48.492205 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 17:57:48.492391 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.488311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:38151 (local address 127.7.118.2:42821)
0929 17:57:48.488439 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:48.488444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:48.488858 (+ 414us) server_negotiation.cc:407] Connection header received
0929 17:57:48.488925 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:48.488928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:48.488980 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:48.489065 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:48.489544 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.490220 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.491409 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.491823 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.491942 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.492006 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:48.492192 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 17:57:48.492260 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 17:57:48.492408 9890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.488232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:45999 (local address 127.7.118.1:46471)
0929 17:57:48.488689 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:48.488704 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:48.488836 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:48.489088 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:48.489091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:48.489132 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:48.489419 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.489428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.490391 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.490393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.491246 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.491256 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.491910 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.491913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.492016 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.492035 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 17:57:48.492122 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 17:57:48.492195 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":101,"spinlock_wait_cycles":6016,"thread_start_us":120,"threads_started":1}
I20260929 17:57:48.492420 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.488232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:42821 (local address 127.7.118.1:38151)
0929 17:57:48.488724 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:48.488739 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:48.488851 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:48.489088 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:48.489091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:48.489131 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:48.489419 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.489428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.490400 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.490404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.491279 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.491290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.491945 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.491948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.492053 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.492080 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 17:57:48.492165 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 17:57:48.492246 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":89,"spinlock_wait_cycles":29184,"thread_start_us":145,"threads_started":1}
I20260929 17:57:48.493302 9811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df719946884e4885b11f9a2cfbeeb8b7" candidate_uuid: "0aaa166ca9684c14a9cf42334976693a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e71099a7b5c462db122ac2a7e033be9" is_pre_election: true
I20260929 17:57:48.493624 9811 raft_consensus.cc:2454] T df719946884e4885b11f9a2cfbeeb8b7 P 9e71099a7b5c462db122ac2a7e033be9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aaa166ca9684c14a9cf42334976693a in term 0.
I20260929 17:57:48.494175 9671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df719946884e4885b11f9a2cfbeeb8b7" candidate_uuid: "0aaa166ca9684c14a9cf42334976693a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "965153f8a69c4e6aa33715bb5a033801" is_pre_election: true
I20260929 17:57:48.494179 9462 leader_election.cc:302] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aaa166ca9684c14a9cf42334976693a, 9e71099a7b5c462db122ac2a7e033be9; no voters:
I20260929 17:57:48.494540 9671 raft_consensus.cc:2454] T df719946884e4885b11f9a2cfbeeb8b7 P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aaa166ca9684c14a9cf42334976693a in term 0.
I20260929 17:57:48.494638 9887 raft_consensus.cc:2781] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:57:48.494766 9887 raft_consensus.cc:482] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:57:48.494879 9887 raft_consensus.cc:3037] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:48.496073 9887 raft_consensus.cc:504] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } } peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } }
I20260929 17:57:48.496518 9887 leader_election.cc:288] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a [CANDIDATE]: Term 1 election: Requested vote from peers 965153f8a69c4e6aa33715bb5a033801 (127.7.118.2:42821), 9e71099a7b5c462db122ac2a7e033be9 (127.7.118.3:45999)
I20260929 17:57:48.496944 9671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df719946884e4885b11f9a2cfbeeb8b7" candidate_uuid: "0aaa166ca9684c14a9cf42334976693a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "965153f8a69c4e6aa33715bb5a033801"
I20260929 17:57:48.496975 9811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df719946884e4885b11f9a2cfbeeb8b7" candidate_uuid: "0aaa166ca9684c14a9cf42334976693a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e71099a7b5c462db122ac2a7e033be9"
I20260929 17:57:48.497064 9671 raft_consensus.cc:3037] T df719946884e4885b11f9a2cfbeeb8b7 P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:48.497148 9811 raft_consensus.cc:3037] T df719946884e4885b11f9a2cfbeeb8b7 P 9e71099a7b5c462db122ac2a7e033be9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:48.498351 9671 raft_consensus.cc:2454] T df719946884e4885b11f9a2cfbeeb8b7 P 965153f8a69c4e6aa33715bb5a033801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aaa166ca9684c14a9cf42334976693a in term 1.
I20260929 17:57:48.498402 9811 raft_consensus.cc:2454] T df719946884e4885b11f9a2cfbeeb8b7 P 9e71099a7b5c462db122ac2a7e033be9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aaa166ca9684c14a9cf42334976693a in term 1.
I20260929 17:57:48.498780 9462 leader_election.cc:302] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aaa166ca9684c14a9cf42334976693a, 9e71099a7b5c462db122ac2a7e033be9; no voters:
I20260929 17:57:48.498986 9887 raft_consensus.cc:2781] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:48.499264 9887 raft_consensus.cc:686] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a [term 1 LEADER]: Becoming Leader. State: Replica: 0aaa166ca9684c14a9cf42334976693a, State: Running, Role: LEADER
I20260929 17:57:48.499701 9887 consensus_queue.cc:237] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } } peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } }
I20260929 17:57:48.503479 9373 catalog_manager.cc:5495] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a reported cstate change: term changed from 0 to 1, leader changed from <none> to 0aaa166ca9684c14a9cf42334976693a (127.7.118.1). New cstate: current_term: 1 leader_uuid: "0aaa166ca9684c14a9cf42334976693a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } health_report { overall_health: HEALTHY } } }
Sep 29 17:57:48 dist-test-slave-r7ph krb5kdc[9329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704667, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:57:48.553934 9898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.544145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:46455 (local address 127.0.0.1:40050)
0929 17:57:48.544510 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:48.544524 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:48.544642 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:48.544955 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:48.544958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:48.545554 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:48.545664 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.545670 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.546650 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.546653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.547295 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.547304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.547870 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.547874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.547950 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.548598 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:48.548612 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:48.550362 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:48.552317 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:48.552328 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:48.552334 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:48.552606 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:48.552907 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:48.552909 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:48.552910 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:48.552990 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:48.553435 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:48.553442 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:48.553554 (+ 112us) client_negotiation.cc:769] Sending connection context
0929 17:57:48.553650 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 17:57:48.553780 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":123,"threads_started":1}
I20260929 17:57:48.554052 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.544340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40050 (local address 127.7.118.62:46455)
0929 17:57:48.544532 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:48.544537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:48.544636 (+ 99us) server_negotiation.cc:407] Connection header received
0929 17:57:48.544729 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:48.544731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:48.544803 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:48.544889 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:48.545800 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.546542 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.547442 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.547694 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.547798 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.550569 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:48.550572 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:48.550575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:48.550600 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:48.552179 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:48.552726 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:48.552729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:48.552730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:48.552814 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:48.553095 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:48.553098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:48.553099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:48.553328 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:48.553414 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:48.553800 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 17:57:48.553917 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 17:57:48.559150 9373 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:40050:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 17:57:48.560050 9373 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:57:48.566085 9511 tablet_service.cc:1467] Processing CreateTablet for tablet 8cf1d42a910a428da819b123f14a524b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=579e871ed5a740dcb69de99dab67ecb2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 17:57:48.566278 9791 tablet_service.cc:1467] Processing CreateTablet for tablet 8cf1d42a910a428da819b123f14a524b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=579e871ed5a740dcb69de99dab67ecb2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 17:57:48.566308 9648 tablet_service.cc:1467] Processing CreateTablet for tablet 8cf1d42a910a428da819b123f14a524b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=579e871ed5a740dcb69de99dab67ecb2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 17:57:48.566994 9791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cf1d42a910a428da819b123f14a524b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:48.566994 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cf1d42a910a428da819b123f14a524b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:48.566998 9648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cf1d42a910a428da819b123f14a524b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:48.569870 9880 tablet_bootstrap.cc:492] T 8cf1d42a910a428da819b123f14a524b P 965153f8a69c4e6aa33715bb5a033801: Bootstrap starting.
I20260929 17:57:48.569928 9882 tablet_bootstrap.cc:492] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a: Bootstrap starting.
I20260929 17:57:48.570461 9881 tablet_bootstrap.cc:492] T 8cf1d42a910a428da819b123f14a524b P 9e71099a7b5c462db122ac2a7e033be9: Bootstrap starting.
I20260929 17:57:48.571017 9880 tablet_bootstrap.cc:654] T 8cf1d42a910a428da819b123f14a524b P 965153f8a69c4e6aa33715bb5a033801: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:48.571022 9882 tablet_bootstrap.cc:654] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:48.571553 9881 tablet_bootstrap.cc:654] T 8cf1d42a910a428da819b123f14a524b P 9e71099a7b5c462db122ac2a7e033be9: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:48.572588 9880 tablet_bootstrap.cc:492] T 8cf1d42a910a428da819b123f14a524b P 965153f8a69c4e6aa33715bb5a033801: No bootstrap required, opened a new log
I20260929 17:57:48.572690 9880 ts_tablet_manager.cc:1397] T 8cf1d42a910a428da819b123f14a524b P 965153f8a69c4e6aa33715bb5a033801: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 17:57:48.572919 9881 tablet_bootstrap.cc:492] T 8cf1d42a910a428da819b123f14a524b P 9e71099a7b5c462db122ac2a7e033be9: No bootstrap required, opened a new log
I20260929 17:57:48.573016 9881 ts_tablet_manager.cc:1397] T 8cf1d42a910a428da819b123f14a524b P 9e71099a7b5c462db122ac2a7e033be9: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 17:57:48.573242 9882 tablet_bootstrap.cc:492] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a: No bootstrap required, opened a new log
I20260929 17:57:48.573227 9880 raft_consensus.cc:348] T 8cf1d42a910a428da819b123f14a524b P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:48.573349 9880 raft_consensus.cc:374] T 8cf1d42a910a428da819b123f14a524b P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:48.573356 9882 ts_tablet_manager.cc:1397] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 17:57:48.573390 9880 raft_consensus.cc:729] T 8cf1d42a910a428da819b123f14a524b P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 965153f8a69c4e6aa33715bb5a033801, State: Initialized, Role: FOLLOWER
I20260929 17:57:48.573493 9881 raft_consensus.cc:348] T 8cf1d42a910a428da819b123f14a524b P 9e71099a7b5c462db122ac2a7e033be9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:48.573535 9880 consensus_queue.cc:260] T 8cf1d42a910a428da819b123f14a524b P 965153f8a69c4e6aa33715bb5a033801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:48.573601 9881 raft_consensus.cc:374] T 8cf1d42a910a428da819b123f14a524b P 9e71099a7b5c462db122ac2a7e033be9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:48.573632 9881 raft_consensus.cc:729] T 8cf1d42a910a428da819b123f14a524b P 9e71099a7b5c462db122ac2a7e033be9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e71099a7b5c462db122ac2a7e033be9, State: Initialized, Role: FOLLOWER
I20260929 17:57:48.573755 9881 consensus_queue.cc:260] T 8cf1d42a910a428da819b123f14a524b P 9e71099a7b5c462db122ac2a7e033be9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:48.573812 9886 tablet_replica.cc:439] T 8cf1d42a910a428da819b123f14a524b P 965153f8a69c4e6aa33715bb5a033801: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } } }
I20260929 17:57:48.573935 9886 tablet_replica.cc:442] T 8cf1d42a910a428da819b123f14a524b P 965153f8a69c4e6aa33715bb5a033801: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:57:48.573861 9882 raft_consensus.cc:348] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:48.573982 9882 raft_consensus.cc:374] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:48.574020 9882 raft_consensus.cc:729] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aaa166ca9684c14a9cf42334976693a, State: Initialized, Role: FOLLOWER
I20260929 17:57:48.574048 9881 ts_tablet_manager.cc:1428] T 8cf1d42a910a428da819b123f14a524b P 9e71099a7b5c462db122ac2a7e033be9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 17:57:48.574156 9882 consensus_queue.cc:260] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:48.574448 9882 ts_tablet_manager.cc:1428] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:57:48.574609 9888 tablet_replica.cc:439] T 8cf1d42a910a428da819b123f14a524b P 9e71099a7b5c462db122ac2a7e033be9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } } }
I20260929 17:57:48.574651 9887 tablet_replica.cc:439] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } } }
I20260929 17:57:48.574770 9887 tablet_replica.cc:442] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:57:48.574772 9888 tablet_replica.cc:442] T 8cf1d42a910a428da819b123f14a524b P 9e71099a7b5c462db122ac2a7e033be9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:57:48.575802 9880 ts_tablet_manager.cc:1428] T 8cf1d42a910a428da819b123f14a524b P 965153f8a69c4e6aa33715bb5a033801: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260929 17:57:48.606628 9578 tablet.cc:2369] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:57:48.611119 9718 tablet.cc:2369] T 8cf1d42a910a428da819b123f14a524b P 965153f8a69c4e6aa33715bb5a033801: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:57:48.631201 9858 tablet.cc:2369] T 8cf1d42a910a428da819b123f14a524b P 9e71099a7b5c462db122ac2a7e033be9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:57:48.725867 9887 raft_consensus.cc:482] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:57:48.726059 9887 raft_consensus.cc:504] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:48.726534 9887 leader_election.cc:288] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 965153f8a69c4e6aa33715bb5a033801 (127.7.118.2:42821), 9e71099a7b5c462db122ac2a7e033be9 (127.7.118.3:45999)
I20260929 17:57:48.727069 9811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cf1d42a910a428da819b123f14a524b" candidate_uuid: "0aaa166ca9684c14a9cf42334976693a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e71099a7b5c462db122ac2a7e033be9" is_pre_election: true
I20260929 17:57:48.727069 9671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cf1d42a910a428da819b123f14a524b" candidate_uuid: "0aaa166ca9684c14a9cf42334976693a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "965153f8a69c4e6aa33715bb5a033801" is_pre_election: true
I20260929 17:57:48.727221 9671 raft_consensus.cc:2454] T 8cf1d42a910a428da819b123f14a524b P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aaa166ca9684c14a9cf42334976693a in term 0.
I20260929 17:57:48.727252 9811 raft_consensus.cc:2454] T 8cf1d42a910a428da819b123f14a524b P 9e71099a7b5c462db122ac2a7e033be9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aaa166ca9684c14a9cf42334976693a in term 0.
I20260929 17:57:48.727603 9462 leader_election.cc:302] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aaa166ca9684c14a9cf42334976693a, 9e71099a7b5c462db122ac2a7e033be9; no voters:
I20260929 17:57:48.727893 9887 raft_consensus.cc:2781] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:57:48.727973 9887 raft_consensus.cc:482] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:57:48.728029 9887 raft_consensus.cc:3037] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:48.729035 9887 raft_consensus.cc:504] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:48.729398 9887 leader_election.cc:288] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a [CANDIDATE]: Term 1 election: Requested vote from peers 965153f8a69c4e6aa33715bb5a033801 (127.7.118.2:42821), 9e71099a7b5c462db122ac2a7e033be9 (127.7.118.3:45999)
I20260929 17:57:48.729907 9811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cf1d42a910a428da819b123f14a524b" candidate_uuid: "0aaa166ca9684c14a9cf42334976693a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e71099a7b5c462db122ac2a7e033be9"
I20260929 17:57:48.729907 9671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cf1d42a910a428da819b123f14a524b" candidate_uuid: "0aaa166ca9684c14a9cf42334976693a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "965153f8a69c4e6aa33715bb5a033801"
I20260929 17:57:48.730049 9671 raft_consensus.cc:3037] T 8cf1d42a910a428da819b123f14a524b P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:48.730049 9811 raft_consensus.cc:3037] T 8cf1d42a910a428da819b123f14a524b P 9e71099a7b5c462db122ac2a7e033be9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:48.731254 9671 raft_consensus.cc:2454] T 8cf1d42a910a428da819b123f14a524b P 965153f8a69c4e6aa33715bb5a033801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aaa166ca9684c14a9cf42334976693a in term 1.
I20260929 17:57:48.731254 9811 raft_consensus.cc:2454] T 8cf1d42a910a428da819b123f14a524b P 9e71099a7b5c462db122ac2a7e033be9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aaa166ca9684c14a9cf42334976693a in term 1.
I20260929 17:57:48.731612 9464 leader_election.cc:302] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aaa166ca9684c14a9cf42334976693a, 965153f8a69c4e6aa33715bb5a033801; no voters:
I20260929 17:57:48.731961 9887 raft_consensus.cc:2781] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:48.732071 9887 raft_consensus.cc:686] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a [term 1 LEADER]: Becoming Leader. State: Replica: 0aaa166ca9684c14a9cf42334976693a, State: Running, Role: LEADER
I20260929 17:57:48.732216 9887 consensus_queue.cc:237] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:48.733026 9891 tablet_replica.cc:439] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a: TxnStatusTablet state changed. Reason: New leader 0aaa166ca9684c14a9cf42334976693a. Latest consensus state: current_term: 1 leader_uuid: "0aaa166ca9684c14a9cf42334976693a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } } }
I20260929 17:57:48.733175 9891 tablet_replica.cc:442] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a: This TxnStatusTablet replica's current role is: LEADER
I20260929 17:57:48.733546 9903 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 17:57:48.733840 9903 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 17:57:48.734432 9373 catalog_manager.cc:5495] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a reported cstate change: term changed from 0 to 1, leader changed from <none> to 0aaa166ca9684c14a9cf42334976693a (127.7.118.1). New cstate: current_term: 1 leader_uuid: "0aaa166ca9684c14a9cf42334976693a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } health_report { overall_health: UNKNOWN } } }
I20260929 17:57:48.769505 9898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.764756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:44893 (local address 127.0.0.1:39892)
0929 17:57:48.765055 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:48.765069 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:48.765225 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:48.765634 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:48.765637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:48.765645 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:57:48.765757 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.765763 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.766857 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.766860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.767533 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.767543 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.768221 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.768224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.768299 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.768352 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:57:48.769200 (+ 848us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:57:48.769218 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 17:57:48.769289 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 17:57:48.769379 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182}
I20260929 17:57:48.769521 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.764897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39892 (local address 127.7.118.1:44893)
0929 17:57:48.765041 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:48.765046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:48.765160 (+ 114us) server_negotiation.cc:407] Connection header received
0929 17:57:48.765410 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:48.765413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:48.765486 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:48.765597 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:57:48.765874 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.766724 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.767746 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.768093 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.768207 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.768505 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:57:48.769084 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:57:48.769213 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:48.769311 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 17:57:48.769380 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 17:57:48.775224 9811 raft_consensus.cc:1261] T 8cf1d42a910a428da819b123f14a524b P 9e71099a7b5c462db122ac2a7e033be9 [term 1 FOLLOWER]: Refusing update from remote peer 0aaa166ca9684c14a9cf42334976693a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:48.775257 9671 raft_consensus.cc:1261] T 8cf1d42a910a428da819b123f14a524b P 965153f8a69c4e6aa33715bb5a033801 [term 1 FOLLOWER]: Refusing update from remote peer 0aaa166ca9684c14a9cf42334976693a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:48.776042 9891 consensus_queue.cc:1035] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a [LEADER]: Connected to new peer: Peer: permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:57:48.776293 9887 consensus_queue.cc:1035] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:57:48.781061 9886 tablet_replica.cc:439] T 8cf1d42a910a428da819b123f14a524b P 965153f8a69c4e6aa33715bb5a033801: TxnStatusTablet state changed. Reason: New leader 0aaa166ca9684c14a9cf42334976693a. Latest consensus state: current_term: 1 leader_uuid: "0aaa166ca9684c14a9cf42334976693a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } } }
I20260929 17:57:48.781060 9888 tablet_replica.cc:439] T 8cf1d42a910a428da819b123f14a524b P 9e71099a7b5c462db122ac2a7e033be9: TxnStatusTablet state changed. Reason: New leader 0aaa166ca9684c14a9cf42334976693a. Latest consensus state: current_term: 1 leader_uuid: "0aaa166ca9684c14a9cf42334976693a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } } }
I20260929 17:57:48.781206 9888 tablet_replica.cc:442] T 8cf1d42a910a428da819b123f14a524b P 9e71099a7b5c462db122ac2a7e033be9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:57:48.781208 9886 tablet_replica.cc:442] T 8cf1d42a910a428da819b123f14a524b P 965153f8a69c4e6aa33715bb5a033801: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:57:48.784518 9887 tablet_replica.cc:439] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0aaa166ca9684c14a9cf42334976693a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } } }
I20260929 17:57:48.784672 9887 tablet_replica.cc:442] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a: This TxnStatusTablet replica's current role is: LEADER
I20260929 17:57:48.784619 9888 tablet_replica.cc:439] T 8cf1d42a910a428da819b123f14a524b P 9e71099a7b5c462db122ac2a7e033be9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0aaa166ca9684c14a9cf42334976693a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } } }
I20260929 17:57:48.784762 9888 tablet_replica.cc:442] T 8cf1d42a910a428da819b123f14a524b P 9e71099a7b5c462db122ac2a7e033be9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:57:48.785526 9891 tablet_replica.cc:439] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0aaa166ca9684c14a9cf42334976693a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } } }
I20260929 17:57:48.785630 9891 tablet_replica.cc:442] T 8cf1d42a910a428da819b123f14a524b P 0aaa166ca9684c14a9cf42334976693a: This TxnStatusTablet replica's current role is: LEADER
I20260929 17:57:48.786242 9908 mvcc.cc:205] Tried to move back new op lower bound from 7334726323295297536 to 7334726323128782848. Current Snapshot: MvccSnapshot[applied={T|T < 7334726323295297536}]
I20260929 17:57:48.786432 9906 mvcc.cc:205] Tried to move back new op lower bound from 7334726323295297536 to 7334726323128782848. Current Snapshot: MvccSnapshot[applied={T|T < 7334726323295297536}]
I20260929 17:57:48.787130 9907 mvcc.cc:205] Tried to move back new op lower bound from 7334726323295297536 to 7334726323128782848. Current Snapshot: MvccSnapshot[applied={T|T < 7334726323295297536}]
I20260929 17:57:48.787374 9886 tablet_replica.cc:439] T 8cf1d42a910a428da819b123f14a524b P 965153f8a69c4e6aa33715bb5a033801: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0aaa166ca9684c14a9cf42334976693a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } } }
I20260929 17:57:48.787473 9886 tablet_replica.cc:442] T 8cf1d42a910a428da819b123f14a524b P 965153f8a69c4e6aa33715bb5a033801: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:57:48.796527 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.792703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39896 (local address 127.7.118.1:44893)
0929 17:57:48.792835 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:48.792840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:48.792857 (+ 17us) server_negotiation.cc:407] Connection header received
0929 17:57:48.792916 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:48.792918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:48.792963 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:48.793039 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:57:48.793364 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.793920 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.795244 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.795527 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.795634 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.795944 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:57:48.796024 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:57:48.796089 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:48.796252 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 17:57:48.796304 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 17:57:48.809490 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.805121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:52743 (local address 127.7.118.1:44893)
0929 17:57:48.805253 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:48.805258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:48.805443 (+ 185us) server_negotiation.cc:407] Connection header received
0929 17:57:48.806120 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:48.806122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:48.806166 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:48.806271 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:57:48.806566 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.807050 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.808283 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.808488 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.808569 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.808949 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:57:48.809044 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:57:48.809130 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:48.809295 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 17:57:48.809355 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 17:57:48.809544 9917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.805048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:44893 (local address 127.7.118.1:52743)
0929 17:57:48.805355 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:48.805369 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:48.806004 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:48.806299 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:48.806302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:48.806311 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:57:48.806460 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.806466 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.807153 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.807156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.808145 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.808156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.808603 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.808606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.808704 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.808765 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:57:48.809135 (+ 370us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:57:48.809142 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 17:57:48.809265 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 17:57:48.809349 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":98,"threads_started":1}
I20260929 17:57:48.817006 9671 raft_consensus.cc:1261] T df719946884e4885b11f9a2cfbeeb8b7 P 965153f8a69c4e6aa33715bb5a033801 [term 1 FOLLOWER]: Refusing update from remote peer 0aaa166ca9684c14a9cf42334976693a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:48.817116 9811 raft_consensus.cc:1261] T df719946884e4885b11f9a2cfbeeb8b7 P 9e71099a7b5c462db122ac2a7e033be9 [term 1 FOLLOWER]: Refusing update from remote peer 0aaa166ca9684c14a9cf42334976693a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:48.817504 9887 consensus_queue.cc:1035] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:57:48.817692 9891 consensus_queue.cc:1035] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a [LEADER]: Connected to new peer: Peer: permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:57:48.866956 9372 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:57:48.867168 9372 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:57:48.870191 9372 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862: Sending DeleteTablet for 3 replicas of tablet df719946884e4885b11f9a2cfbeeb8b7
I20260929 17:57:48.871356 9648 tablet_service.cc:1514] Processing DeleteTablet for tablet df719946884e4885b11f9a2cfbeeb8b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:48 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:42154
I20260929 17:57:48.871361 9791 tablet_service.cc:1514] Processing DeleteTablet for tablet df719946884e4885b11f9a2cfbeeb8b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:48 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:41716
I20260929 17:57:48.872066 9507 tablet_service.cc:1514] Processing DeleteTablet for tablet df719946884e4885b11f9a2cfbeeb8b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:48 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:39880
I20260929 17:57:48.872640 9925 tablet_replica.cc:330] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a: stopping tablet replica
I20260929 17:57:48.872900 9372 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 17:57:48.873042 9925 raft_consensus.cc:2229] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a [term 1 LEADER]: Raft consensus shutting down.
W20260929 17:57:48.873380 9372 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:57:48.873793 9925 raft_consensus.cc:2258] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:57:48.875553 9925 ts_tablet_manager.cc:1905] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:57:48.878664 9507 tablet_service.cc:1467] Processing CreateTablet for tablet 603d167b952a49a5b8262c67738ea543 (DEFAULT_TABLE table=test-table [id=2cf6277e434841a29584b48ff4807183]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:48.878993 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 603d167b952a49a5b8262c67738ea543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:48.879020 9791 tablet_service.cc:1467] Processing CreateTablet for tablet 603d167b952a49a5b8262c67738ea543 (DEFAULT_TABLE table=test-table [id=2cf6277e434841a29584b48ff4807183]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:48.879333 9791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 603d167b952a49a5b8262c67738ea543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:48.879760 9925 ts_tablet_manager.cc:1918] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:57:48.879916 9925 log.cc:1196] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-0/wal/wals/df719946884e4885b11f9a2cfbeeb8b7
I20260929 17:57:48.880456 9925 ts_tablet_manager.cc:1939] T df719946884e4885b11f9a2cfbeeb8b7 P 0aaa166ca9684c14a9cf42334976693a: Deleting consensus metadata
I20260929 17:57:48.881552 9882 tablet_bootstrap.cc:492] T 603d167b952a49a5b8262c67738ea543 P 0aaa166ca9684c14a9cf42334976693a: Bootstrap starting.
I20260929 17:57:48.882627 9882 tablet_bootstrap.cc:654] T 603d167b952a49a5b8262c67738ea543 P 0aaa166ca9684c14a9cf42334976693a: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:48.882709 9648 tablet_service.cc:1467] Processing CreateTablet for tablet 603d167b952a49a5b8262c67738ea543 (DEFAULT_TABLE table=test-table [id=2cf6277e434841a29584b48ff4807183]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:48.883025 9648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 603d167b952a49a5b8262c67738ea543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:48.883474 9358 catalog_manager.cc:4841] TS 0aaa166ca9684c14a9cf42334976693a (127.7.118.1:44893): tablet df719946884e4885b11f9a2cfbeeb8b7 (table test-table [id=ecfaa6977e40477aba1c4fd51a74b796]) successfully deleted
I20260929 17:57:48.884090 9882 tablet_bootstrap.cc:492] T 603d167b952a49a5b8262c67738ea543 P 0aaa166ca9684c14a9cf42334976693a: No bootstrap required, opened a new log
I20260929 17:57:48.884186 9882 ts_tablet_manager.cc:1397] T 603d167b952a49a5b8262c67738ea543 P 0aaa166ca9684c14a9cf42334976693a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 17:57:48.884769 9882 raft_consensus.cc:348] T 603d167b952a49a5b8262c67738ea543 P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:48.884905 9882 raft_consensus.cc:374] T 603d167b952a49a5b8262c67738ea543 P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:48.884939 9882 raft_consensus.cc:729] T 603d167b952a49a5b8262c67738ea543 P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aaa166ca9684c14a9cf42334976693a, State: Initialized, Role: FOLLOWER
I20260929 17:57:48.885082 9882 consensus_queue.cc:260] T 603d167b952a49a5b8262c67738ea543 P 0aaa166ca9684c14a9cf42334976693a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:48.885360 9882 ts_tablet_manager.cc:1428] T 603d167b952a49a5b8262c67738ea543 P 0aaa166ca9684c14a9cf42334976693a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:57:48.885700 9881 tablet_bootstrap.cc:492] T 603d167b952a49a5b8262c67738ea543 P 9e71099a7b5c462db122ac2a7e033be9: Bootstrap starting.
I20260929 17:57:48.886808 9881 tablet_bootstrap.cc:654] T 603d167b952a49a5b8262c67738ea543 P 9e71099a7b5c462db122ac2a7e033be9: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:48.888355 9881 tablet_bootstrap.cc:492] T 603d167b952a49a5b8262c67738ea543 P 9e71099a7b5c462db122ac2a7e033be9: No bootstrap required, opened a new log
I20260929 17:57:48.888474 9881 ts_tablet_manager.cc:1397] T 603d167b952a49a5b8262c67738ea543 P 9e71099a7b5c462db122ac2a7e033be9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 17:57:48.889009 9881 raft_consensus.cc:348] T 603d167b952a49a5b8262c67738ea543 P 9e71099a7b5c462db122ac2a7e033be9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:48.889156 9881 raft_consensus.cc:374] T 603d167b952a49a5b8262c67738ea543 P 9e71099a7b5c462db122ac2a7e033be9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:48.889215 9881 raft_consensus.cc:729] T 603d167b952a49a5b8262c67738ea543 P 9e71099a7b5c462db122ac2a7e033be9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e71099a7b5c462db122ac2a7e033be9, State: Initialized, Role: FOLLOWER
I20260929 17:57:48.890291 9880 tablet_bootstrap.cc:492] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801: Bootstrap starting.
I20260929 17:57:48.890368 9881 consensus_queue.cc:260] T 603d167b952a49a5b8262c67738ea543 P 9e71099a7b5c462db122ac2a7e033be9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:48.891149 9881 ts_tablet_manager.cc:1428] T 603d167b952a49a5b8262c67738ea543 P 9e71099a7b5c462db122ac2a7e033be9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 17:57:48.891436 9880 tablet_bootstrap.cc:654] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:48.891460 9924 tablet_replica.cc:330] T df719946884e4885b11f9a2cfbeeb8b7 P 965153f8a69c4e6aa33715bb5a033801: stopping tablet replica
I20260929 17:57:48.891827 9924 raft_consensus.cc:2229] T df719946884e4885b11f9a2cfbeeb8b7 P 965153f8a69c4e6aa33715bb5a033801 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:57:48.892091 9924 raft_consensus.cc:2258] T df719946884e4885b11f9a2cfbeeb8b7 P 965153f8a69c4e6aa33715bb5a033801 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:57:48.893834 9924 ts_tablet_manager.cc:1905] T df719946884e4885b11f9a2cfbeeb8b7 P 965153f8a69c4e6aa33715bb5a033801: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:57:48.894141 9880 tablet_bootstrap.cc:492] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801: No bootstrap required, opened a new log
I20260929 17:57:48.894229 9880 ts_tablet_manager.cc:1397] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 17:57:48.894801 9880 raft_consensus.cc:348] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:48.894927 9880 raft_consensus.cc:374] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:48.894959 9880 raft_consensus.cc:729] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 965153f8a69c4e6aa33715bb5a033801, State: Initialized, Role: FOLLOWER
I20260929 17:57:48.895123 9880 consensus_queue.cc:260] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:48.896842 9880 ts_tablet_manager.cc:1428] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260929 17:57:48.897333 9924 ts_tablet_manager.cc:1918] T df719946884e4885b11f9a2cfbeeb8b7 P 965153f8a69c4e6aa33715bb5a033801: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:57:48.897452 9924 log.cc:1196] T df719946884e4885b11f9a2cfbeeb8b7 P 965153f8a69c4e6aa33715bb5a033801: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-1/wal/wals/df719946884e4885b11f9a2cfbeeb8b7
I20260929 17:57:48.897763 9924 ts_tablet_manager.cc:1939] T df719946884e4885b11f9a2cfbeeb8b7 P 965153f8a69c4e6aa33715bb5a033801: Deleting consensus metadata
I20260929 17:57:48.899142 9360 catalog_manager.cc:4841] TS 965153f8a69c4e6aa33715bb5a033801 (127.7.118.2:42821): tablet df719946884e4885b11f9a2cfbeeb8b7 (table test-table [id=ecfaa6977e40477aba1c4fd51a74b796]) successfully deleted
I20260929 17:57:48.899219 9923 tablet_replica.cc:330] T df719946884e4885b11f9a2cfbeeb8b7 P 9e71099a7b5c462db122ac2a7e033be9: stopping tablet replica
I20260929 17:57:48.899592 9923 raft_consensus.cc:2229] T df719946884e4885b11f9a2cfbeeb8b7 P 9e71099a7b5c462db122ac2a7e033be9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:57:48.899852 9923 raft_consensus.cc:2258] T df719946884e4885b11f9a2cfbeeb8b7 P 9e71099a7b5c462db122ac2a7e033be9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:57:48.901326 9923 ts_tablet_manager.cc:1905] T df719946884e4885b11f9a2cfbeeb8b7 P 9e71099a7b5c462db122ac2a7e033be9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:57:48.904003 9923 ts_tablet_manager.cc:1918] T df719946884e4885b11f9a2cfbeeb8b7 P 9e71099a7b5c462db122ac2a7e033be9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:57:48.904114 9923 log.cc:1196] T df719946884e4885b11f9a2cfbeeb8b7 P 9e71099a7b5c462db122ac2a7e033be9: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-2/wal/wals/df719946884e4885b11f9a2cfbeeb8b7
I20260929 17:57:48.904505 9923 ts_tablet_manager.cc:1939] T df719946884e4885b11f9a2cfbeeb8b7 P 9e71099a7b5c462db122ac2a7e033be9: Deleting consensus metadata
I20260929 17:57:48.905787 9358 catalog_manager.cc:4841] TS 9e71099a7b5c462db122ac2a7e033be9 (127.7.118.3:45999): tablet df719946884e4885b11f9a2cfbeeb8b7 (table test-table [id=ecfaa6977e40477aba1c4fd51a74b796]) successfully deleted
I20260929 17:57:48.922508 9886 raft_consensus.cc:482] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:57:48.922698 9886 raft_consensus.cc:504] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:48.924013 9886 leader_election.cc:288] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0aaa166ca9684c14a9cf42334976693a (127.7.118.1:44893), 9e71099a7b5c462db122ac2a7e033be9 (127.7.118.3:45999)
I20260929 17:57:48.928768 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.924351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:57593 (local address 127.7.118.1:44893)
0929 17:57:48.924502 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:48.924508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:48.924593 (+ 85us) server_negotiation.cc:407] Connection header received
0929 17:57:48.924843 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:48.924847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:48.924906 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:48.924992 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:48.925299 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.925953 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.927471 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.928131 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.928264 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.928335 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:48.928545 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 17:57:48.928632 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 17:57:48.928783 9929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.924240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:44893 (local address 127.7.118.2:57593)
0929 17:57:48.924503 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:48.924520 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:48.924685 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:48.925020 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:48.925026 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:48.925050 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:48.925200 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.925209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.926057 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.926062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.927320 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.927330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.928281 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.928285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.928409 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.928433 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 17:57:48.928511 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 17:57:48.928602 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":81,"threads_started":1}
I20260929 17:57:48.928828 9877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.924735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:56115 (local address 127.7.118.3:45999)
0929 17:57:48.924865 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:48.924869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:48.925065 (+ 196us) server_negotiation.cc:407] Connection header received
0929 17:57:48.925248 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:48.925252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:48.925308 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:48.925387 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:48.925655 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.926451 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.927741 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.928150 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.928275 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.928341 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:48.928633 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 17:57:48.928715 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 17:57:48.929247 9930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.924614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:45999 (local address 127.7.118.2:56115)
0929 17:57:48.924979 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:48.924993 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:48.925116 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:48.925405 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:48.925408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:48.925426 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:48.925532 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.925538 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.926568 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.926571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.927621 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.927628 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.928409 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.928412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.928508 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.928527 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 17:57:48.929005 (+ 478us) client_negotiation.cc:240] Negotiation successful
0929 17:57:48.929109 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":10,"thread_start_us":52,"threads_started":1}
I20260929 17:57:48.929459 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603d167b952a49a5b8262c67738ea543" candidate_uuid: "965153f8a69c4e6aa33715bb5a033801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aaa166ca9684c14a9cf42334976693a" is_pre_election: true
I20260929 17:57:48.929721 9531 raft_consensus.cc:2454] T 603d167b952a49a5b8262c67738ea543 P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 965153f8a69c4e6aa33715bb5a033801 in term 0.
I20260929 17:57:48.930194 9602 leader_election.cc:302] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aaa166ca9684c14a9cf42334976693a, 965153f8a69c4e6aa33715bb5a033801; no voters:
I20260929 17:57:48.930321 9810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603d167b952a49a5b8262c67738ea543" candidate_uuid: "965153f8a69c4e6aa33715bb5a033801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e71099a7b5c462db122ac2a7e033be9" is_pre_election: true
I20260929 17:57:48.930481 9810 raft_consensus.cc:2454] T 603d167b952a49a5b8262c67738ea543 P 9e71099a7b5c462db122ac2a7e033be9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 965153f8a69c4e6aa33715bb5a033801 in term 0.
I20260929 17:57:48.930521 9886 raft_consensus.cc:2781] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:57:48.930593 9886 raft_consensus.cc:482] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:57:48.930640 9886 raft_consensus.cc:3037] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:48.931926 9886 raft_consensus.cc:504] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:48.932289 9886 leader_election.cc:288] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801 [CANDIDATE]: Term 1 election: Requested vote from peers 0aaa166ca9684c14a9cf42334976693a (127.7.118.1:44893), 9e71099a7b5c462db122ac2a7e033be9 (127.7.118.3:45999)
I20260929 17:57:48.932669 9810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603d167b952a49a5b8262c67738ea543" candidate_uuid: "965153f8a69c4e6aa33715bb5a033801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e71099a7b5c462db122ac2a7e033be9"
I20260929 17:57:48.932734 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603d167b952a49a5b8262c67738ea543" candidate_uuid: "965153f8a69c4e6aa33715bb5a033801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aaa166ca9684c14a9cf42334976693a"
I20260929 17:57:48.932830 9810 raft_consensus.cc:3037] T 603d167b952a49a5b8262c67738ea543 P 9e71099a7b5c462db122ac2a7e033be9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:48.932850 9531 raft_consensus.cc:3037] T 603d167b952a49a5b8262c67738ea543 P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:48.933821 9810 raft_consensus.cc:2454] T 603d167b952a49a5b8262c67738ea543 P 9e71099a7b5c462db122ac2a7e033be9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 965153f8a69c4e6aa33715bb5a033801 in term 1.
I20260929 17:57:48.934005 9531 raft_consensus.cc:2454] T 603d167b952a49a5b8262c67738ea543 P 0aaa166ca9684c14a9cf42334976693a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 965153f8a69c4e6aa33715bb5a033801 in term 1.
I20260929 17:57:48.934122 9602 leader_election.cc:302] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 965153f8a69c4e6aa33715bb5a033801, 9e71099a7b5c462db122ac2a7e033be9; no voters:
I20260929 17:57:48.934346 9886 raft_consensus.cc:2781] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:48.934585 9886 raft_consensus.cc:686] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801 [term 1 LEADER]: Becoming Leader. State: Replica: 965153f8a69c4e6aa33715bb5a033801, State: Running, Role: LEADER
I20260929 17:57:48.934859 9886 consensus_queue.cc:237] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:48.937413 9373 catalog_manager.cc:5495] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801 reported cstate change: term changed from 0 to 1, leader changed from <none> to 965153f8a69c4e6aa33715bb5a033801 (127.7.118.2). New cstate: current_term: 1 leader_uuid: "965153f8a69c4e6aa33715bb5a033801" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } health_report { overall_health: UNKNOWN } } }
I20260929 17:57:48.957777 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.953259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42162 (local address 127.7.118.2:42821)
0929 17:57:48.953389 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:48.953393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:48.953456 (+ 63us) server_negotiation.cc:407] Connection header received
0929 17:57:48.953667 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:48.953670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:48.953787 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:48.953896 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:57:48.954169 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.954834 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.956145 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.956426 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.956547 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.956843 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:57:48.957320 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:57:48.957411 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:48.957546 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 17:57:48.957603 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 17:57:48.969934 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.965787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:44893 (local address 127.7.118.2:60501)
0929 17:57:48.966111 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:48.966127 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:48.966229 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:48.966550 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:48.966552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:48.966559 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:57:48.966708 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.966713 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.967835 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.967839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.968464 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.968473 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.969131 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.969135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.969218 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.969269 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:57:48.969643 (+ 374us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:57:48.969650 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 17:57:48.969714 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 17:57:48.969794 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":130,"threads_started":1}
I20260929 17:57:48.969965 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.965897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:60501 (local address 127.7.118.1:44893)
0929 17:57:48.966034 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:48.966039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:48.966239 (+ 200us) server_negotiation.cc:407] Connection header received
0929 17:57:48.966379 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:48.966383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:48.966440 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:48.966531 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:57:48.966824 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.967691 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.968589 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.968970 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.969086 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:48.969419 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:57:48.969515 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:57:48.969618 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:48.969751 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 17:57:48.969817 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 17:57:48.977248 9810 raft_consensus.cc:1261] T 603d167b952a49a5b8262c67738ea543 P 9e71099a7b5c462db122ac2a7e033be9 [term 1 FOLLOWER]: Refusing update from remote peer 965153f8a69c4e6aa33715bb5a033801: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:48.977413 9531 raft_consensus.cc:1261] T 603d167b952a49a5b8262c67738ea543 P 0aaa166ca9684c14a9cf42334976693a [term 1 FOLLOWER]: Refusing update from remote peer 965153f8a69c4e6aa33715bb5a033801: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:48.977804 9931 consensus_queue.cc:1035] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:57:48.977990 9886 consensus_queue.cc:1035] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:57:49.001439 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.997506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:41499 (local address 127.7.118.2:42821)
0929 17:57:48.997673 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:48.997679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:48.997719 (+ 40us) server_negotiation.cc:407] Connection header received
0929 17:57:48.997863 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:48.997866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:48.997915 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:48.998029 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:57:48.998342 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.999103 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:49.000111 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:49.000452 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:49.000559 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:49.000930 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:57:49.001023 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:57:49.001112 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:49.001222 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 17:57:49.001279 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 17:57:49.001439 9917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:48.997351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:42821 (local address 127.7.118.1:41499)
0929 17:57:48.997656 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:48.997672 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:48.997779 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:48.998041 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:48.998044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:48.998053 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:57:48.998171 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.998181 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:48.999232 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:48.999239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:48.999958 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:48.999971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:49.000669 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:49.000673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:49.000757 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:49.000787 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:57:49.001122 (+ 335us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:57:49.001132 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 17:57:49.001211 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 17:57:49.001313 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
W20260929 17:57:49.002377 9907 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 17:57:50.573269 9372 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:57:50.573438 9372 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:57:50.576135 9372 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862: Sending DeleteTablet for 3 replicas of tablet 603d167b952a49a5b8262c67738ea543
I20260929 17:57:50.576887 9507 tablet_service.cc:1514] Processing DeleteTablet for tablet 603d167b952a49a5b8262c67738ea543 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:50 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:39880
I20260929 17:57:50.576942 9791 tablet_service.cc:1514] Processing DeleteTablet for tablet 603d167b952a49a5b8262c67738ea543 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:50 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:41716
I20260929 17:57:50.577011 9648 tablet_service.cc:1514] Processing DeleteTablet for tablet 603d167b952a49a5b8262c67738ea543 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:50 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:42154
I20260929 17:57:50.578442 9963 tablet_replica.cc:330] T 603d167b952a49a5b8262c67738ea543 P 0aaa166ca9684c14a9cf42334976693a: stopping tablet replica
I20260929 17:57:50.578605 9963 raft_consensus.cc:2229] T 603d167b952a49a5b8262c67738ea543 P 0aaa166ca9684c14a9cf42334976693a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:57:50.578753 9963 raft_consensus.cc:2258] T 603d167b952a49a5b8262c67738ea543 P 0aaa166ca9684c14a9cf42334976693a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:57:50.578867 9962 tablet_replica.cc:330] T 603d167b952a49a5b8262c67738ea543 P 9e71099a7b5c462db122ac2a7e033be9: stopping tablet replica
I20260929 17:57:50.579052 9962 raft_consensus.cc:2229] T 603d167b952a49a5b8262c67738ea543 P 9e71099a7b5c462db122ac2a7e033be9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:57:50.579110 9964 tablet_replica.cc:330] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801: stopping tablet replica
I20260929 17:57:50.579276 9962 raft_consensus.cc:2258] T 603d167b952a49a5b8262c67738ea543 P 9e71099a7b5c462db122ac2a7e033be9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:57:50.579284 9964 raft_consensus.cc:2229] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:57:50.579515 9963 ts_tablet_manager.cc:1905] T 603d167b952a49a5b8262c67738ea543 P 0aaa166ca9684c14a9cf42334976693a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:57:50.579892 9964 raft_consensus.cc:2258] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:57:50.580081 9962 ts_tablet_manager.cc:1905] T 603d167b952a49a5b8262c67738ea543 P 9e71099a7b5c462db122ac2a7e033be9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:57:50.580700 9964 ts_tablet_manager.cc:1905] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:57:50.582667 9963 ts_tablet_manager.cc:1918] T 603d167b952a49a5b8262c67738ea543 P 0aaa166ca9684c14a9cf42334976693a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 17:57:50.582772 9963 log.cc:1196] T 603d167b952a49a5b8262c67738ea543 P 0aaa166ca9684c14a9cf42334976693a: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-0/wal/wals/603d167b952a49a5b8262c67738ea543
I20260929 17:57:50.583038 9963 ts_tablet_manager.cc:1939] T 603d167b952a49a5b8262c67738ea543 P 0aaa166ca9684c14a9cf42334976693a: Deleting consensus metadata
I20260929 17:57:50.583021 9962 ts_tablet_manager.cc:1918] T 603d167b952a49a5b8262c67738ea543 P 9e71099a7b5c462db122ac2a7e033be9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 17:57:50.583215 9962 log.cc:1196] T 603d167b952a49a5b8262c67738ea543 P 9e71099a7b5c462db122ac2a7e033be9: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-2/wal/wals/603d167b952a49a5b8262c67738ea543
I20260929 17:57:50.583539 9962 ts_tablet_manager.cc:1939] T 603d167b952a49a5b8262c67738ea543 P 9e71099a7b5c462db122ac2a7e033be9: Deleting consensus metadata
I20260929 17:57:50.584057 9964 ts_tablet_manager.cc:1918] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 17:57:50.584174 9964 log.cc:1196] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-1/wal/wals/603d167b952a49a5b8262c67738ea543
I20260929 17:57:50.584157 9358 catalog_manager.cc:4841] TS 0aaa166ca9684c14a9cf42334976693a (127.7.118.1:44893): tablet 603d167b952a49a5b8262c67738ea543 (table test-table [id=2cf6277e434841a29584b48ff4807183]) successfully deleted
I20260929 17:57:50.584374 9358 catalog_manager.cc:4841] TS 9e71099a7b5c462db122ac2a7e033be9 (127.7.118.3:45999): tablet 603d167b952a49a5b8262c67738ea543 (table test-table [id=2cf6277e434841a29584b48ff4807183]) successfully deleted
I20260929 17:57:50.584491 9964 ts_tablet_manager.cc:1939] T 603d167b952a49a5b8262c67738ea543 P 965153f8a69c4e6aa33715bb5a033801: Deleting consensus metadata
I20260929 17:57:50.585271 9360 catalog_manager.cc:4841] TS 965153f8a69c4e6aa33715bb5a033801 (127.7.118.2:42821): tablet 603d167b952a49a5b8262c67738ea543 (table test-table [id=2cf6277e434841a29584b48ff4807183]) successfully deleted
Sep 29 17:57:50 dist-test-slave-r7ph krb5kdc[9329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704670, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 17:57:50 dist-test-slave-r7ph krb5kdc[9329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704670, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:57:50.607453 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:50.597685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40062 (local address 127.7.118.62:46455)
0929 17:57:50.597971 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:50.597978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:50.598030 (+ 52us) server_negotiation.cc:407] Connection header received
0929 17:57:50.598167 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:50.598170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:50.598213 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:50.598355 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:50.599074 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:50.599619 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:50.600632 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:50.600916 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:50.601002 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:50.603883 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:50.603887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:50.603889 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:50.603922 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:50.605693 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:50.606218 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:50.606220 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:50.606222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:50.606379 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:50.606675 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:50.606678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:50.606679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:50.606867 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:50.606972 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:50.607210 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 17:57:50.607318 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":137,"threads_started":1}
I20260929 17:57:50.610765 9372 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:57:50.611276 9372 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:57:50.616380 9507 tablet_service.cc:1467] Processing CreateTablet for tablet 16d9f938471140199ad5c06663e0c903 (DEFAULT_TABLE table=test-table [id=3ae72c626b4b4c9494ed308fbac4e757]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:50.616552 9791 tablet_service.cc:1467] Processing CreateTablet for tablet 16d9f938471140199ad5c06663e0c903 (DEFAULT_TABLE table=test-table [id=3ae72c626b4b4c9494ed308fbac4e757]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:50.616732 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16d9f938471140199ad5c06663e0c903. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:50.616849 9791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16d9f938471140199ad5c06663e0c903. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:50.616811 9648 tablet_service.cc:1467] Processing CreateTablet for tablet 16d9f938471140199ad5c06663e0c903 (DEFAULT_TABLE table=test-table [id=3ae72c626b4b4c9494ed308fbac4e757]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:50.617054 9648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16d9f938471140199ad5c06663e0c903. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:50.619263 9973 tablet_bootstrap.cc:492] T 16d9f938471140199ad5c06663e0c903 P 0aaa166ca9684c14a9cf42334976693a: Bootstrap starting.
I20260929 17:57:50.619819 9975 tablet_bootstrap.cc:492] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801: Bootstrap starting.
I20260929 17:57:50.620113 9973 tablet_bootstrap.cc:654] T 16d9f938471140199ad5c06663e0c903 P 0aaa166ca9684c14a9cf42334976693a: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:50.620676 9975 tablet_bootstrap.cc:654] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:50.621284 9973 tablet_bootstrap.cc:492] T 16d9f938471140199ad5c06663e0c903 P 0aaa166ca9684c14a9cf42334976693a: No bootstrap required, opened a new log
I20260929 17:57:50.621342 9974 tablet_bootstrap.cc:492] T 16d9f938471140199ad5c06663e0c903 P 9e71099a7b5c462db122ac2a7e033be9: Bootstrap starting.
I20260929 17:57:50.621454 9973 ts_tablet_manager.cc:1397] T 16d9f938471140199ad5c06663e0c903 P 0aaa166ca9684c14a9cf42334976693a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 17:57:50.621989 9975 tablet_bootstrap.cc:492] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801: No bootstrap required, opened a new log
I20260929 17:57:50.622093 9975 ts_tablet_manager.cc:1397] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 17:57:50.622397 9974 tablet_bootstrap.cc:654] T 16d9f938471140199ad5c06663e0c903 P 9e71099a7b5c462db122ac2a7e033be9: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:50.622655 9975 raft_consensus.cc:348] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:50.622709 9973 raft_consensus.cc:348] T 16d9f938471140199ad5c06663e0c903 P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:50.622793 9975 raft_consensus.cc:374] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:50.622833 9975 raft_consensus.cc:729] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 965153f8a69c4e6aa33715bb5a033801, State: Initialized, Role: FOLLOWER
I20260929 17:57:50.622840 9973 raft_consensus.cc:374] T 16d9f938471140199ad5c06663e0c903 P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:50.622885 9973 raft_consensus.cc:729] T 16d9f938471140199ad5c06663e0c903 P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aaa166ca9684c14a9cf42334976693a, State: Initialized, Role: FOLLOWER
I20260929 17:57:50.622975 9975 consensus_queue.cc:260] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:50.623045 9973 consensus_queue.cc:260] T 16d9f938471140199ad5c06663e0c903 P 0aaa166ca9684c14a9cf42334976693a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:50.623314 9975 ts_tablet_manager.cc:1428] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:57:50.623836 9973 ts_tablet_manager.cc:1428] T 16d9f938471140199ad5c06663e0c903 P 0aaa166ca9684c14a9cf42334976693a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 17:57:50.623970 9974 tablet_bootstrap.cc:492] T 16d9f938471140199ad5c06663e0c903 P 9e71099a7b5c462db122ac2a7e033be9: No bootstrap required, opened a new log
I20260929 17:57:50.624074 9974 ts_tablet_manager.cc:1397] T 16d9f938471140199ad5c06663e0c903 P 9e71099a7b5c462db122ac2a7e033be9: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 17:57:50.624552 9974 raft_consensus.cc:348] T 16d9f938471140199ad5c06663e0c903 P 9e71099a7b5c462db122ac2a7e033be9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:50.624682 9974 raft_consensus.cc:374] T 16d9f938471140199ad5c06663e0c903 P 9e71099a7b5c462db122ac2a7e033be9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:50.624715 9974 raft_consensus.cc:729] T 16d9f938471140199ad5c06663e0c903 P 9e71099a7b5c462db122ac2a7e033be9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e71099a7b5c462db122ac2a7e033be9, State: Initialized, Role: FOLLOWER
I20260929 17:57:50.624864 9974 consensus_queue.cc:260] T 16d9f938471140199ad5c06663e0c903 P 9e71099a7b5c462db122ac2a7e033be9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:50.625270 9974 ts_tablet_manager.cc:1428] T 16d9f938471140199ad5c06663e0c903 P 9e71099a7b5c462db122ac2a7e033be9: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260929 17:57:50.626345 9956 raft_consensus.cc:482] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:57:50.626432 9956 raft_consensus.cc:504] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:50.626755 9956 leader_election.cc:288] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0aaa166ca9684c14a9cf42334976693a (127.7.118.1:44893), 9e71099a7b5c462db122ac2a7e033be9 (127.7.118.3:45999)
I20260929 17:57:50.627199 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d9f938471140199ad5c06663e0c903" candidate_uuid: "965153f8a69c4e6aa33715bb5a033801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aaa166ca9684c14a9cf42334976693a" is_pre_election: true
I20260929 17:57:50.627341 9531 raft_consensus.cc:2454] T 16d9f938471140199ad5c06663e0c903 P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 965153f8a69c4e6aa33715bb5a033801 in term 0.
I20260929 17:57:50.627637 9810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d9f938471140199ad5c06663e0c903" candidate_uuid: "965153f8a69c4e6aa33715bb5a033801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e71099a7b5c462db122ac2a7e033be9" is_pre_election: true
I20260929 17:57:50.627748 9810 raft_consensus.cc:2454] T 16d9f938471140199ad5c06663e0c903 P 9e71099a7b5c462db122ac2a7e033be9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 965153f8a69c4e6aa33715bb5a033801 in term 0.
I20260929 17:57:50.627724 9602 leader_election.cc:302] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aaa166ca9684c14a9cf42334976693a, 965153f8a69c4e6aa33715bb5a033801; no voters:
I20260929 17:57:50.627897 9956 raft_consensus.cc:2781] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:57:50.627952 9956 raft_consensus.cc:482] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:57:50.628000 9956 raft_consensus.cc:3037] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:50.629258 9956 raft_consensus.cc:504] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:50.629621 9956 leader_election.cc:288] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801 [CANDIDATE]: Term 1 election: Requested vote from peers 0aaa166ca9684c14a9cf42334976693a (127.7.118.1:44893), 9e71099a7b5c462db122ac2a7e033be9 (127.7.118.3:45999)
I20260929 17:57:50.629982 9810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d9f938471140199ad5c06663e0c903" candidate_uuid: "965153f8a69c4e6aa33715bb5a033801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e71099a7b5c462db122ac2a7e033be9"
I20260929 17:57:50.629974 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d9f938471140199ad5c06663e0c903" candidate_uuid: "965153f8a69c4e6aa33715bb5a033801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aaa166ca9684c14a9cf42334976693a"
I20260929 17:57:50.630100 9531 raft_consensus.cc:3037] T 16d9f938471140199ad5c06663e0c903 P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:50.630129 9810 raft_consensus.cc:3037] T 16d9f938471140199ad5c06663e0c903 P 9e71099a7b5c462db122ac2a7e033be9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:50.631270 9531 raft_consensus.cc:2454] T 16d9f938471140199ad5c06663e0c903 P 0aaa166ca9684c14a9cf42334976693a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 965153f8a69c4e6aa33715bb5a033801 in term 1.
I20260929 17:57:50.631270 9810 raft_consensus.cc:2454] T 16d9f938471140199ad5c06663e0c903 P 9e71099a7b5c462db122ac2a7e033be9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 965153f8a69c4e6aa33715bb5a033801 in term 1.
I20260929 17:57:50.631563 9602 leader_election.cc:302] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aaa166ca9684c14a9cf42334976693a, 965153f8a69c4e6aa33715bb5a033801; no voters:
I20260929 17:57:50.631744 9956 raft_consensus.cc:2781] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:50.631850 9956 raft_consensus.cc:686] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801 [term 1 LEADER]: Becoming Leader. State: Replica: 965153f8a69c4e6aa33715bb5a033801, State: Running, Role: LEADER
I20260929 17:57:50.632014 9956 consensus_queue.cc:237] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:50.633685 9371 catalog_manager.cc:5495] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801 reported cstate change: term changed from 0 to 1, leader changed from <none> to 965153f8a69c4e6aa33715bb5a033801 (127.7.118.2). New cstate: current_term: 1 leader_uuid: "965153f8a69c4e6aa33715bb5a033801" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } health_report { overall_health: UNKNOWN } } }
I20260929 17:57:50.653297 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:50.649578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42172 (local address 127.7.118.2:42821)
0929 17:57:50.649943 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:50.649950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:50.649964 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:57:50.650005 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:50.650007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:50.650049 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:50.650124 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:57:50.650474 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:50.651084 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:50.652059 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:50.652320 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:50.652415 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:50.652753 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:57:50.652838 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:57:50.652902 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:50.653005 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 17:57:50.653056 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":129,"threads_started":1}
I20260929 17:57:50.660374 9810 raft_consensus.cc:1261] T 16d9f938471140199ad5c06663e0c903 P 9e71099a7b5c462db122ac2a7e033be9 [term 1 FOLLOWER]: Refusing update from remote peer 965153f8a69c4e6aa33715bb5a033801: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:50.660516 9531 raft_consensus.cc:1261] T 16d9f938471140199ad5c06663e0c903 P 0aaa166ca9684c14a9cf42334976693a [term 1 FOLLOWER]: Refusing update from remote peer 965153f8a69c4e6aa33715bb5a033801: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:50.660820 9956 consensus_queue.cc:1035] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:57:50.660955 9953 consensus_queue.cc:1035] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:57:50.690788 9371 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:57:50.690938 9371 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:57:50.692934 9371 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862: Sending DeleteTablet for 3 replicas of tablet 16d9f938471140199ad5c06663e0c903
I20260929 17:57:50.693706 9791 tablet_service.cc:1514] Processing DeleteTablet for tablet 16d9f938471140199ad5c06663e0c903 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:50 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:41716
I20260929 17:57:50.693753 9648 tablet_service.cc:1514] Processing DeleteTablet for tablet 16d9f938471140199ad5c06663e0c903 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:50 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:42154
I20260929 17:57:50.693957 9962 tablet_replica.cc:330] T 16d9f938471140199ad5c06663e0c903 P 9e71099a7b5c462db122ac2a7e033be9: stopping tablet replica
I20260929 17:57:50.693969 9964 tablet_replica.cc:330] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801: stopping tablet replica
I20260929 17:57:50.694088 9962 raft_consensus.cc:2229] T 16d9f938471140199ad5c06663e0c903 P 9e71099a7b5c462db122ac2a7e033be9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:57:50.694113 9964 raft_consensus.cc:2229] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:57:50.694240 9962 raft_consensus.cc:2258] T 16d9f938471140199ad5c06663e0c903 P 9e71099a7b5c462db122ac2a7e033be9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:57:50.694387 9964 raft_consensus.cc:2258] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:57:50.695003 9507 tablet_service.cc:1514] Processing DeleteTablet for tablet 16d9f938471140199ad5c06663e0c903 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:50 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:39880
I20260929 17:57:50.695045 9962 ts_tablet_manager.cc:1905] T 16d9f938471140199ad5c06663e0c903 P 9e71099a7b5c462db122ac2a7e033be9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:57:50.695226 9963 tablet_replica.cc:330] T 16d9f938471140199ad5c06663e0c903 P 0aaa166ca9684c14a9cf42334976693a: stopping tablet replica
I20260929 17:57:50.695374 9963 raft_consensus.cc:2229] T 16d9f938471140199ad5c06663e0c903 P 0aaa166ca9684c14a9cf42334976693a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:57:50.695497 9963 raft_consensus.cc:2258] T 16d9f938471140199ad5c06663e0c903 P 0aaa166ca9684c14a9cf42334976693a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:57:50.695401 9371 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 17:57:50.695657 9964 ts_tablet_manager.cc:1905] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 17:57:50.695868 9371 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:57:50.697793 9962 ts_tablet_manager.cc:1918] T 16d9f938471140199ad5c06663e0c903 P 9e71099a7b5c462db122ac2a7e033be9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:57:50.697863 9962 log.cc:1196] T 16d9f938471140199ad5c06663e0c903 P 9e71099a7b5c462db122ac2a7e033be9: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-2/wal/wals/16d9f938471140199ad5c06663e0c903
I20260929 17:57:50.698171 9962 ts_tablet_manager.cc:1939] T 16d9f938471140199ad5c06663e0c903 P 9e71099a7b5c462db122ac2a7e033be9: Deleting consensus metadata
I20260929 17:57:50.698233 9964 ts_tablet_manager.cc:1918] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:57:50.698344 9964 log.cc:1196] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-1/wal/wals/16d9f938471140199ad5c06663e0c903
I20260929 17:57:50.698654 9964 ts_tablet_manager.cc:1939] T 16d9f938471140199ad5c06663e0c903 P 965153f8a69c4e6aa33715bb5a033801: Deleting consensus metadata
I20260929 17:57:50.699123 9358 catalog_manager.cc:4841] TS 9e71099a7b5c462db122ac2a7e033be9 (127.7.118.3:45999): tablet 16d9f938471140199ad5c06663e0c903 (table test-table [id=3ae72c626b4b4c9494ed308fbac4e757]) successfully deleted
I20260929 17:57:50.699294 9360 catalog_manager.cc:4841] TS 965153f8a69c4e6aa33715bb5a033801 (127.7.118.2:42821): tablet 16d9f938471140199ad5c06663e0c903 (table test-table [id=3ae72c626b4b4c9494ed308fbac4e757]) successfully deleted
I20260929 17:57:50.700162 9963 ts_tablet_manager.cc:1905] T 16d9f938471140199ad5c06663e0c903 P 0aaa166ca9684c14a9cf42334976693a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:57:50.701498 9791 tablet_service.cc:1467] Processing CreateTablet for tablet d501945f70e14f62962a4064853798b1 (DEFAULT_TABLE table=test-table [id=954fe7fae18746e2921a01e753f6211e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:50.701498 9507 tablet_service.cc:1467] Processing CreateTablet for tablet d501945f70e14f62962a4064853798b1 (DEFAULT_TABLE table=test-table [id=954fe7fae18746e2921a01e753f6211e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:50.701778 9791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d501945f70e14f62962a4064853798b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:50.701778 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d501945f70e14f62962a4064853798b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:50.703174 9963 ts_tablet_manager.cc:1918] T 16d9f938471140199ad5c06663e0c903 P 0aaa166ca9684c14a9cf42334976693a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:57:50.703282 9963 log.cc:1196] T 16d9f938471140199ad5c06663e0c903 P 0aaa166ca9684c14a9cf42334976693a: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-0/wal/wals/16d9f938471140199ad5c06663e0c903
I20260929 17:57:50.703524 9963 ts_tablet_manager.cc:1939] T 16d9f938471140199ad5c06663e0c903 P 0aaa166ca9684c14a9cf42334976693a: Deleting consensus metadata
I20260929 17:57:50.704136 9973 tablet_bootstrap.cc:492] T d501945f70e14f62962a4064853798b1 P 0aaa166ca9684c14a9cf42334976693a: Bootstrap starting.
I20260929 17:57:50.704237 9974 tablet_bootstrap.cc:492] T d501945f70e14f62962a4064853798b1 P 9e71099a7b5c462db122ac2a7e033be9: Bootstrap starting.
I20260929 17:57:50.704241 9648 tablet_service.cc:1467] Processing CreateTablet for tablet d501945f70e14f62962a4064853798b1 (DEFAULT_TABLE table=test-table [id=954fe7fae18746e2921a01e753f6211e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:50.704537 9648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d501945f70e14f62962a4064853798b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:50.705070 9973 tablet_bootstrap.cc:654] T d501945f70e14f62962a4064853798b1 P 0aaa166ca9684c14a9cf42334976693a: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:50.705080 9974 tablet_bootstrap.cc:654] T d501945f70e14f62962a4064853798b1 P 9e71099a7b5c462db122ac2a7e033be9: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:50.706019 9358 catalog_manager.cc:4841] TS 0aaa166ca9684c14a9cf42334976693a (127.7.118.1:44893): tablet 16d9f938471140199ad5c06663e0c903 (table test-table [id=3ae72c626b4b4c9494ed308fbac4e757]) successfully deleted
I20260929 17:57:50.706321 9973 tablet_bootstrap.cc:492] T d501945f70e14f62962a4064853798b1 P 0aaa166ca9684c14a9cf42334976693a: No bootstrap required, opened a new log
I20260929 17:57:50.706413 9973 ts_tablet_manager.cc:1397] T d501945f70e14f62962a4064853798b1 P 0aaa166ca9684c14a9cf42334976693a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 17:57:50.706847 9973 raft_consensus.cc:348] T d501945f70e14f62962a4064853798b1 P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:50.706951 9975 tablet_bootstrap.cc:492] T d501945f70e14f62962a4064853798b1 P 965153f8a69c4e6aa33715bb5a033801: Bootstrap starting.
I20260929 17:57:50.706972 9973 raft_consensus.cc:374] T d501945f70e14f62962a4064853798b1 P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:50.707010 9973 raft_consensus.cc:729] T d501945f70e14f62962a4064853798b1 P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aaa166ca9684c14a9cf42334976693a, State: Initialized, Role: FOLLOWER
I20260929 17:57:50.707134 9973 consensus_queue.cc:260] T d501945f70e14f62962a4064853798b1 P 0aaa166ca9684c14a9cf42334976693a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:50.707401 9973 ts_tablet_manager.cc:1428] T d501945f70e14f62962a4064853798b1 P 0aaa166ca9684c14a9cf42334976693a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:57:50.707863 9975 tablet_bootstrap.cc:654] T d501945f70e14f62962a4064853798b1 P 965153f8a69c4e6aa33715bb5a033801: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:50.707886 9974 tablet_bootstrap.cc:492] T d501945f70e14f62962a4064853798b1 P 9e71099a7b5c462db122ac2a7e033be9: No bootstrap required, opened a new log
I20260929 17:57:50.707970 9974 ts_tablet_manager.cc:1397] T d501945f70e14f62962a4064853798b1 P 9e71099a7b5c462db122ac2a7e033be9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260929 17:57:50.708482 9974 raft_consensus.cc:348] T d501945f70e14f62962a4064853798b1 P 9e71099a7b5c462db122ac2a7e033be9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:50.708607 9974 raft_consensus.cc:374] T d501945f70e14f62962a4064853798b1 P 9e71099a7b5c462db122ac2a7e033be9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:50.708647 9974 raft_consensus.cc:729] T d501945f70e14f62962a4064853798b1 P 9e71099a7b5c462db122ac2a7e033be9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e71099a7b5c462db122ac2a7e033be9, State: Initialized, Role: FOLLOWER
I20260929 17:57:50.708796 9974 consensus_queue.cc:260] T d501945f70e14f62962a4064853798b1 P 9e71099a7b5c462db122ac2a7e033be9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:50.709085 9974 ts_tablet_manager.cc:1428] T d501945f70e14f62962a4064853798b1 P 9e71099a7b5c462db122ac2a7e033be9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 17:57:50.709986 9975 tablet_bootstrap.cc:492] T d501945f70e14f62962a4064853798b1 P 965153f8a69c4e6aa33715bb5a033801: No bootstrap required, opened a new log
I20260929 17:57:50.710086 9975 ts_tablet_manager.cc:1397] T d501945f70e14f62962a4064853798b1 P 965153f8a69c4e6aa33715bb5a033801: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 17:57:50.710598 9975 raft_consensus.cc:348] T d501945f70e14f62962a4064853798b1 P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:50.710718 9975 raft_consensus.cc:374] T d501945f70e14f62962a4064853798b1 P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:50.710755 9975 raft_consensus.cc:729] T d501945f70e14f62962a4064853798b1 P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 965153f8a69c4e6aa33715bb5a033801, State: Initialized, Role: FOLLOWER
I20260929 17:57:50.710886 9975 consensus_queue.cc:260] T d501945f70e14f62962a4064853798b1 P 965153f8a69c4e6aa33715bb5a033801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:50.711192 9975 ts_tablet_manager.cc:1428] T d501945f70e14f62962a4064853798b1 P 965153f8a69c4e6aa33715bb5a033801: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:57:50.743171 9943 raft_consensus.cc:482] T d501945f70e14f62962a4064853798b1 P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:57:50.743320 9943 raft_consensus.cc:504] T d501945f70e14f62962a4064853798b1 P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:50.743889 9943 leader_election.cc:288] T d501945f70e14f62962a4064853798b1 P 0aaa166ca9684c14a9cf42334976693a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 965153f8a69c4e6aa33715bb5a033801 (127.7.118.2:42821), 9e71099a7b5c462db122ac2a7e033be9 (127.7.118.3:45999)
I20260929 17:57:50.744397 9811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d501945f70e14f62962a4064853798b1" candidate_uuid: "0aaa166ca9684c14a9cf42334976693a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e71099a7b5c462db122ac2a7e033be9" is_pre_election: true
I20260929 17:57:50.744400 9671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d501945f70e14f62962a4064853798b1" candidate_uuid: "0aaa166ca9684c14a9cf42334976693a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "965153f8a69c4e6aa33715bb5a033801" is_pre_election: true
I20260929 17:57:50.744536 9811 raft_consensus.cc:2454] T d501945f70e14f62962a4064853798b1 P 9e71099a7b5c462db122ac2a7e033be9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aaa166ca9684c14a9cf42334976693a in term 0.
I20260929 17:57:50.744596 9671 raft_consensus.cc:2454] T d501945f70e14f62962a4064853798b1 P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aaa166ca9684c14a9cf42334976693a in term 0.
I20260929 17:57:50.744966 9462 leader_election.cc:302] T d501945f70e14f62962a4064853798b1 P 0aaa166ca9684c14a9cf42334976693a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aaa166ca9684c14a9cf42334976693a, 9e71099a7b5c462db122ac2a7e033be9; no voters:
I20260929 17:57:50.745193 9943 raft_consensus.cc:2781] T d501945f70e14f62962a4064853798b1 P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:57:50.745256 9943 raft_consensus.cc:482] T d501945f70e14f62962a4064853798b1 P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:57:50.745307 9943 raft_consensus.cc:3037] T d501945f70e14f62962a4064853798b1 P 0aaa166ca9684c14a9cf42334976693a [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:50.746673 9943 raft_consensus.cc:504] T d501945f70e14f62962a4064853798b1 P 0aaa166ca9684c14a9cf42334976693a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:50.747119 9943 leader_election.cc:288] T d501945f70e14f62962a4064853798b1 P 0aaa166ca9684c14a9cf42334976693a [CANDIDATE]: Term 1 election: Requested vote from peers 965153f8a69c4e6aa33715bb5a033801 (127.7.118.2:42821), 9e71099a7b5c462db122ac2a7e033be9 (127.7.118.3:45999)
I20260929 17:57:50.747517 9811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d501945f70e14f62962a4064853798b1" candidate_uuid: "0aaa166ca9684c14a9cf42334976693a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e71099a7b5c462db122ac2a7e033be9"
I20260929 17:57:50.747534 9671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d501945f70e14f62962a4064853798b1" candidate_uuid: "0aaa166ca9684c14a9cf42334976693a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "965153f8a69c4e6aa33715bb5a033801"
I20260929 17:57:50.747668 9671 raft_consensus.cc:3037] T d501945f70e14f62962a4064853798b1 P 965153f8a69c4e6aa33715bb5a033801 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:50.747668 9811 raft_consensus.cc:3037] T d501945f70e14f62962a4064853798b1 P 9e71099a7b5c462db122ac2a7e033be9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:50.748862 9811 raft_consensus.cc:2454] T d501945f70e14f62962a4064853798b1 P 9e71099a7b5c462db122ac2a7e033be9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aaa166ca9684c14a9cf42334976693a in term 1.
I20260929 17:57:50.748863 9671 raft_consensus.cc:2454] T d501945f70e14f62962a4064853798b1 P 965153f8a69c4e6aa33715bb5a033801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aaa166ca9684c14a9cf42334976693a in term 1.
I20260929 17:57:50.749248 9462 leader_election.cc:302] T d501945f70e14f62962a4064853798b1 P 0aaa166ca9684c14a9cf42334976693a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0aaa166ca9684c14a9cf42334976693a, 965153f8a69c4e6aa33715bb5a033801, 9e71099a7b5c462db122ac2a7e033be9; no voters:
I20260929 17:57:50.749449 9943 raft_consensus.cc:2781] T d501945f70e14f62962a4064853798b1 P 0aaa166ca9684c14a9cf42334976693a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:50.749543 9943 raft_consensus.cc:686] T d501945f70e14f62962a4064853798b1 P 0aaa166ca9684c14a9cf42334976693a [term 1 LEADER]: Becoming Leader. State: Replica: 0aaa166ca9684c14a9cf42334976693a, State: Running, Role: LEADER
I20260929 17:57:50.749702 9943 consensus_queue.cc:237] T d501945f70e14f62962a4064853798b1 P 0aaa166ca9684c14a9cf42334976693a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } }
I20260929 17:57:50.751552 9374 catalog_manager.cc:5495] T d501945f70e14f62962a4064853798b1 P 0aaa166ca9684c14a9cf42334976693a reported cstate change: term changed from 0 to 1, leader changed from <none> to 0aaa166ca9684c14a9cf42334976693a (127.7.118.1). New cstate: current_term: 1 leader_uuid: "0aaa166ca9684c14a9cf42334976693a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aaa166ca9684c14a9cf42334976693a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 44893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 } health_report { overall_health: UNKNOWN } } }
I20260929 17:57:50.772490 9990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:50.768697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39908 (local address 127.7.118.1:44893)
0929 17:57:50.769057 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:50.769064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:50.769077 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:57:50.769122 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:50.769124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:50.769167 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:50.769253 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:57:50.769647 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:50.770191 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:50.771324 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:50.771584 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:50.771704 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:50.771966 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:57:50.772052 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:57:50.772126 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:50.772298 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 17:57:50.772360 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":151,"threads_started":1}
I20260929 17:57:50.780313 9811 raft_consensus.cc:1261] T d501945f70e14f62962a4064853798b1 P 9e71099a7b5c462db122ac2a7e033be9 [term 1 FOLLOWER]: Refusing update from remote peer 0aaa166ca9684c14a9cf42334976693a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:50.780406 9671 raft_consensus.cc:1261] T d501945f70e14f62962a4064853798b1 P 965153f8a69c4e6aa33715bb5a033801 [term 1 FOLLOWER]: Refusing update from remote peer 0aaa166ca9684c14a9cf42334976693a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:50.780755 9943 consensus_queue.cc:1035] T d501945f70e14f62962a4064853798b1 P 0aaa166ca9684c14a9cf42334976693a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e71099a7b5c462db122ac2a7e033be9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:57:50.780921 9948 consensus_queue.cc:1035] T d501945f70e14f62962a4064853798b1 P 0aaa166ca9684c14a9cf42334976693a [LEADER]: Connected to new peer: Peer: permanent_uuid: "965153f8a69c4e6aa33715bb5a033801" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 17:57:50.796195 9941 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 17:57:52.351042 9373 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:57:52.351259 9373 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:57:52.354429 9373 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7008ea9982e4f7eaa7ef48b63036862: Sending DeleteTablet for 3 replicas of tablet d501945f70e14f62962a4064853798b1
I20260929 17:57:52.355280 9507 tablet_service.cc:1514] Processing DeleteTablet for tablet d501945f70e14f62962a4064853798b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:52 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:39880
I20260929 17:57:52.355280 9648 tablet_service.cc:1514] Processing DeleteTablet for tablet d501945f70e14f62962a4064853798b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:52 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:42154
I20260929 17:57:52.355309 9791 tablet_service.cc:1514] Processing DeleteTablet for tablet d501945f70e14f62962a4064853798b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:52 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:41716
I20260929 17:57:52.355737 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 9446
I20260929 17:57:52.355778 10016 tablet_replica.cc:330] T d501945f70e14f62962a4064853798b1 P 9e71099a7b5c462db122ac2a7e033be9: stopping tablet replica
I20260929 17:57:52.356546 10016 raft_consensus.cc:2229] T d501945f70e14f62962a4064853798b1 P 9e71099a7b5c462db122ac2a7e033be9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:57:52.356663 10014 tablet_replica.cc:330] T d501945f70e14f62962a4064853798b1 P 965153f8a69c4e6aa33715bb5a033801: stopping tablet replica
I20260929 17:57:52.356750 10016 raft_consensus.cc:2258] T d501945f70e14f62962a4064853798b1 P 9e71099a7b5c462db122ac2a7e033be9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:57:52.356814 10014 raft_consensus.cc:2229] T d501945f70e14f62962a4064853798b1 P 965153f8a69c4e6aa33715bb5a033801 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:57:52.356947 10014 raft_consensus.cc:2258] T d501945f70e14f62962a4064853798b1 P 965153f8a69c4e6aa33715bb5a033801 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:57:52.357836 10014 ts_tablet_manager.cc:1905] T d501945f70e14f62962a4064853798b1 P 965153f8a69c4e6aa33715bb5a033801: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:57:52.357954 10016 ts_tablet_manager.cc:1905] T d501945f70e14f62962a4064853798b1 P 9e71099a7b5c462db122ac2a7e033be9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:57:52.361282 10016 ts_tablet_manager.cc:1918] T d501945f70e14f62962a4064853798b1 P 9e71099a7b5c462db122ac2a7e033be9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 17:57:52.361414 10016 log.cc:1196] T d501945f70e14f62962a4064853798b1 P 9e71099a7b5c462db122ac2a7e033be9: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-2/wal/wals/d501945f70e14f62962a4064853798b1
I20260929 17:57:52.361704 10016 ts_tablet_manager.cc:1939] T d501945f70e14f62962a4064853798b1 P 9e71099a7b5c462db122ac2a7e033be9: Deleting consensus metadata
I20260929 17:57:52.361743 10014 ts_tablet_manager.cc:1918] T d501945f70e14f62962a4064853798b1 P 965153f8a69c4e6aa33715bb5a033801: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 17:57:52.361831 10014 log.cc:1196] T d501945f70e14f62962a4064853798b1 P 965153f8a69c4e6aa33715bb5a033801: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704657566865-7640-0/minicluster-data/ts-1/wal/wals/d501945f70e14f62962a4064853798b1
I20260929 17:57:52.362118 10014 ts_tablet_manager.cc:1939] T d501945f70e14f62962a4064853798b1 P 965153f8a69c4e6aa33715bb5a033801: Deleting consensus metadata
I20260929 17:57:52.362756 9358 catalog_manager.cc:4841] TS 9e71099a7b5c462db122ac2a7e033be9 (127.7.118.3:45999): tablet d501945f70e14f62962a4064853798b1 (table test-table [id=954fe7fae18746e2921a01e753f6211e]) successfully deleted
I20260929 17:57:52.363016 9360 catalog_manager.cc:4841] TS 965153f8a69c4e6aa33715bb5a033801 (127.7.118.2:42821): tablet d501945f70e14f62962a4064853798b1 (table test-table [id=954fe7fae18746e2921a01e753f6211e]) successfully deleted
W20260929 17:57:52.364997 9358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.118.1:44893 (error 108)
I20260929 17:57:52.365087 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 9586
I20260929 17:57:52.367372 10018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:52.366891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:44893 (local address 127.0.0.1:39914)
0929 17:57:52.367190 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:52.367269 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.1:44893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":119,"threads_started":1}
W20260929 17:57:52.367575 9358 catalog_manager.cc:4584] TS 0aaa166ca9684c14a9cf42334976693a (127.7.118.1:44893): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d501945f70e14f62962a4064853798b1: Network error: Client connection negotiation failed: client connection to 127.7.118.1:44893: connect: Connection refused (error 111)
I20260929 17:57:52.373055 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 9726
I20260929 17:57:52.379624 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 9342
2026-09-29T17:57:52Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6279 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:52 dist-test-slave-r7ph krb5kdc[10021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:52 dist-test-slave-r7ph krb5kdc[10021](info): set up 2 sockets
Sep 29 17:57:52 dist-test-slave-r7ph krb5kdc[10021](info): commencing operation
krb5kdc: starting...
W20260929 17:57:53.498569 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 17:57:53 dist-test-slave-r7ph krb5kdc[10021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 17:57:53.635228 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:37087
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:38367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:37087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:53.756127 10034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:53.756515 10034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:53.756608 10034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:53.761011 10034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:53.761144 10034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:53.761180 10034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:53.761219 10034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:53.761256 10034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:53.767057 10034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:38367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:37087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:37087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:53.768291 10034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:53.769423 10034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:57:53.779575 10034 server_base.cc:1034] running on GCE node
W20260929 17:57:53.779687 10039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:53.779526 10040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:53.779527 10042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:53.780654 10034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:53.781942 10034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:53.783155 10034 hybrid_clock.cc:648] HybridClock initialized: now 1790704673783138 us; error 36 us; skew 500 ppm
Sep 29 17:57:53 dist-test-slave-r7ph krb5kdc[10021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704673, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:53.786860 10034 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 17:57:53.788005 10034 webserver.cc:460] Webserver started at http://127.7.118.62:45615/ using document root <none> and password file <none>
I20260929 17:57:53.788781 10034 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:53.788864 10034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:53.789117 10034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:53.791527 10034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "407c05ad9a3f43beae0bd9beae3c9675"
format_stamp: "Formatted at 2026-09-29 17:57:53 on dist-test-slave-r7ph"
server_key: "a370c626ed48ec146856fb6d1eb01e7f"
server_key_iv: "ad1292cf77fdfeffaf1f794643cf1d0b"
server_key_version: "encryptionkey@0"
I20260929 17:57:53.792205 10034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "407c05ad9a3f43beae0bd9beae3c9675"
format_stamp: "Formatted at 2026-09-29 17:57:53 on dist-test-slave-r7ph"
server_key: "a370c626ed48ec146856fb6d1eb01e7f"
server_key_iv: "ad1292cf77fdfeffaf1f794643cf1d0b"
server_key_version: "encryptionkey@0"
I20260929 17:57:53.796659 10034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:57:53.799762 10049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:53.801435 10034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:57:53.801574 10034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "407c05ad9a3f43beae0bd9beae3c9675"
format_stamp: "Formatted at 2026-09-29 17:57:53 on dist-test-slave-r7ph"
server_key: "a370c626ed48ec146856fb6d1eb01e7f"
server_key_iv: "ad1292cf77fdfeffaf1f794643cf1d0b"
server_key_version: "encryptionkey@0"
I20260929 17:57:53.801694 10034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:53.817728 10034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:53.821521 10034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:53.821712 10034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:53.829804 10034 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:37087
I20260929 17:57:53.829842 10101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:37087 every 8 connection(s)
I20260929 17:57:53.830927 10034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:53.832885 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 10034
I20260929 17:57:53.832986 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 17:57:53.833281 7640 external_mini_cluster.cc:1428] Setting key 895aec0cc762c63e427cd147349a3455
I20260929 17:57:53.835491 10102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:53.842362 10102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 407c05ad9a3f43beae0bd9beae3c9675: Bootstrap starting.
Sep 29 17:57:53 dist-test-slave-r7ph krb5kdc[10021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:57:53.845248 10102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 407c05ad9a3f43beae0bd9beae3c9675: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:53.846128 10102 log.cc:824] T 00000000000000000000000000000000 P 407c05ad9a3f43beae0bd9beae3c9675: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:53.848510 10102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 407c05ad9a3f43beae0bd9beae3c9675: No bootstrap required, opened a new log
I20260929 17:57:53.848734 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:53.834781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41118 (local address 127.7.118.62:37087)
0929 17:57:53.835257 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:53.835273 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:53.835311 (+ 38us) server_negotiation.cc:407] Connection header received
0929 17:57:53.836043 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:53.836064 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:53.836445 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:53.836824 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:53.837665 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:53.838802 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:53.839673 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:53.840171 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:53.840342 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:53.843173 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:53.843179 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:53.843196 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:53.843228 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:53.845299 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:53.845871 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:53.845877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:53.845884 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:53.846019 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:53.846370 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:53.846374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:53.846376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:53.846820 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:53.847042 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:53.847355 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 17:57:53.847661 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":165,"threads_started":1}
I20260929 17:57:53.853984 10102 raft_consensus.cc:348] T 00000000000000000000000000000000 P 407c05ad9a3f43beae0bd9beae3c9675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "407c05ad9a3f43beae0bd9beae3c9675" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 37087 } }
I20260929 17:57:53.854353 10102 raft_consensus.cc:374] T 00000000000000000000000000000000 P 407c05ad9a3f43beae0bd9beae3c9675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:53.854454 10102 raft_consensus.cc:729] T 00000000000000000000000000000000 P 407c05ad9a3f43beae0bd9beae3c9675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 407c05ad9a3f43beae0bd9beae3c9675, State: Initialized, Role: FOLLOWER
I20260929 17:57:53.854978 10102 consensus_queue.cc:260] T 00000000000000000000000000000000 P 407c05ad9a3f43beae0bd9beae3c9675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "407c05ad9a3f43beae0bd9beae3c9675" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 37087 } }
I20260929 17:57:53.855168 10102 raft_consensus.cc:388] T 00000000000000000000000000000000 P 407c05ad9a3f43beae0bd9beae3c9675 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:53.855289 10102 raft_consensus.cc:482] T 00000000000000000000000000000000 P 407c05ad9a3f43beae0bd9beae3c9675 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:53.855430 10102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 407c05ad9a3f43beae0bd9beae3c9675 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:53.856654 10102 raft_consensus.cc:504] T 00000000000000000000000000000000 P 407c05ad9a3f43beae0bd9beae3c9675 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "407c05ad9a3f43beae0bd9beae3c9675" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 37087 } }
I20260929 17:57:53.857087 10102 leader_election.cc:302] T 00000000000000000000000000000000 P 407c05ad9a3f43beae0bd9beae3c9675 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 407c05ad9a3f43beae0bd9beae3c9675; no voters:
I20260929 17:57:53.857570 10102 leader_election.cc:288] T 00000000000000000000000000000000 P 407c05ad9a3f43beae0bd9beae3c9675 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:53.857757 10107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 407c05ad9a3f43beae0bd9beae3c9675 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:53.858060 10107 raft_consensus.cc:686] T 00000000000000000000000000000000 P 407c05ad9a3f43beae0bd9beae3c9675 [term 1 LEADER]: Becoming Leader. State: Replica: 407c05ad9a3f43beae0bd9beae3c9675, State: Running, Role: LEADER
I20260929 17:57:53.858464 10107 consensus_queue.cc:237] T 00000000000000000000000000000000 P 407c05ad9a3f43beae0bd9beae3c9675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "407c05ad9a3f43beae0bd9beae3c9675" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 37087 } }
I20260929 17:57:53.859089 10102 sys_catalog.cc:564] T 00000000000000000000000000000000 P 407c05ad9a3f43beae0bd9beae3c9675 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:53.860486 10108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 407c05ad9a3f43beae0bd9beae3c9675 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 407c05ad9a3f43beae0bd9beae3c9675. Latest consensus state: current_term: 1 leader_uuid: "407c05ad9a3f43beae0bd9beae3c9675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "407c05ad9a3f43beae0bd9beae3c9675" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 37087 } } }
I20260929 17:57:53.860508 10109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 407c05ad9a3f43beae0bd9beae3c9675 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "407c05ad9a3f43beae0bd9beae3c9675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "407c05ad9a3f43beae0bd9beae3c9675" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 37087 } } }
I20260929 17:57:53.860850 10108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 407c05ad9a3f43beae0bd9beae3c9675 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:53.860870 10109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 407c05ad9a3f43beae0bd9beae3c9675 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:53.861404 10118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:53.864678 10118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:53.871791 10118 catalog_manager.cc:1347] Generated new cluster ID: d33dacbe956943959352c1f9007dd98a
I20260929 17:57:53.871896 10118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:53.883574 10118 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:53.884678 10118 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:53.893742 10118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 407c05ad9a3f43beae0bd9beae3c9675: Generated new TSK 0
I20260929 17:57:53.894625 10118 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.118.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
I20260929 17:57:53.935261 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:0
--local_ip_for_outbound_sockets=127.7.118.1
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:37087
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:38367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:54.090102 10128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:54.090500 10128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:54.090550 10128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:54.094906 10128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:54.094993 10128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:54.095058 10128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 17:57:54.101306 10128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:38367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:37087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:54.102551 10128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:54.103847 10128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:57:54.114925 10128 server_base.cc:1034] running on GCE node
W20260929 17:57:54.114868 10134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:54.114864 10136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:54.114864 10133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:54.115770 10128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:54.116609 10128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:54.117848 10128 hybrid_clock.cc:648] HybridClock initialized: now 1790704674117811 us; error 54 us; skew 500 ppm
Sep 29 17:57:54 dist-test-slave-r7ph krb5kdc[10021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704674, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:54.121162 10128 init.cc:377] Logged in from keytab as kudu/127.7.118.1@KRBTEST.COM (short username kudu)
I20260929 17:57:54.122375 10128 webserver.cc:460] Webserver started at http://127.7.118.1:40279/ using document root <none> and password file <none>
I20260929 17:57:54.123005 10128 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:54.123104 10128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:54.123364 10128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:54.125696 10128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/data/instance:
uuid: "19877e5de4994e89a0c918091e305f6d"
format_stamp: "Formatted at 2026-09-29 17:57:54 on dist-test-slave-r7ph"
server_key: "e30f3c5ad9072d7c855e416d9585ae85"
server_key_iv: "1bc667201bffdec9167a3f0a034aa8fd"
server_key_version: "encryptionkey@0"
I20260929 17:57:54.126427 10128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance:
uuid: "19877e5de4994e89a0c918091e305f6d"
format_stamp: "Formatted at 2026-09-29 17:57:54 on dist-test-slave-r7ph"
server_key: "e30f3c5ad9072d7c855e416d9585ae85"
server_key_iv: "1bc667201bffdec9167a3f0a034aa8fd"
server_key_version: "encryptionkey@0"
I20260929 17:57:54.130880 10128 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 17:57:54.133941 10143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:54.135699 10128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:57:54.135892 10128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "19877e5de4994e89a0c918091e305f6d"
format_stamp: "Formatted at 2026-09-29 17:57:54 on dist-test-slave-r7ph"
server_key: "e30f3c5ad9072d7c855e416d9585ae85"
server_key_iv: "1bc667201bffdec9167a3f0a034aa8fd"
server_key_version: "encryptionkey@0"
I20260929 17:57:54.136049 10128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:54.182004 10128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:54.185950 10128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:54.186213 10128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:54.186925 10128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:57:54.187862 10128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:57:54.187937 10128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:54.188019 10128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:57:54.188071 10128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:54.197749 10128 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:39781
I20260929 17:57:54.197767 10256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:39781 every 8 connection(s)
I20260929 17:57:54.198797 10128 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 17:57:54.205813 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 10128
I20260929 17:57:54.205921 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance
I20260929 17:57:54.206354 7640 external_mini_cluster.cc:1428] Setting key c9251670f32d0756af746b47bfaf84af
Sep 29 17:57:54 dist-test-slave-r7ph krb5kdc[10021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704674, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.118.2@KRBTEST.COM; defaulting to no policy
I20260929 17:57:54.216602 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:54.200915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:34663 (local address 127.7.118.62:37087)
0929 17:57:54.201083 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:54.201087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:54.202040 (+ 953us) server_negotiation.cc:407] Connection header received
0929 17:57:54.203576 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:54.203579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:54.203642 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:54.203763 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:54.205117 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:54.205723 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:54.207482 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:54.207739 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:54.207871 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:54.210860 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:54.210864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:54.210866 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:54.210899 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:54.213260 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:54.214208 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:54.214211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:54.214213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:54.214373 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:54.214915 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:54.214918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:54.214919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:54.215286 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:54.215414 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:54.216275 (+ 861us) server_negotiation.cc:299] Negotiation successful
0929 17:57:54.216446 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 17:57:54.217576 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:54.201349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:37087 (local address 127.7.118.1:34663)
0929 17:57:54.201865 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:54.201916 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:54.203305 (+ 1389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:54.203964 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:54.203972 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:54.204396 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:54.204932 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:54.204946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:54.205865 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:54.205891 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:54.207351 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:54.207361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:54.207971 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:54.207974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:54.208156 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:54.208860 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:54.208879 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:54.210557 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:54.213512 (+ 2955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:54.213523 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:54.213549 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:54.214060 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:54.214480 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:54.214485 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:54.214487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:54.214749 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:54.215425 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:54.215435 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:54.215877 (+ 442us) client_negotiation.cc:769] Sending connection context
0929 17:57:54.216149 (+ 272us) client_negotiation.cc:240] Negotiation successful
0929 17:57:54.216584 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":146,"threads_started":1}
I20260929 17:57:54.219328 10257 heartbeater.cc:347] Connected to a master server at 127.7.118.62:37087
I20260929 17:57:54.219694 10257 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.2@KRBTEST.COM" created.
I20260929 17:57:54.220392 10257 heartbeater.cc:511] Master 127.7.118.62:37087 requested a full tablet report, sending...
I20260929 17:57:54.222455 10066 ts_manager.cc:194] Registered new tserver with Master: 19877e5de4994e89a0c918091e305f6d (127.7.118.1:39781)
I20260929 17:57:54.224117 10066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.1@KRBTEST.COM'} at 127.7.118.1:34663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
I20260929 17:57:54.238346 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:0
--local_ip_for_outbound_sockets=127.7.118.2
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:37087
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:38367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:54.358880 10262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:54.359181 10262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:54.359228 10262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:54.363927 10262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:54.364010 10262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:54.364079 10262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 17:57:54.369848 10262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:38367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:37087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:54.371054 10262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:54.372078 10262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:54.382745 10270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:54.382772 10267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:54.382754 10268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:54.383113 10262 server_base.cc:1034] running on GCE node
I20260929 17:57:54.383708 10262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:54.384491 10262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:54.385756 10262 hybrid_clock.cc:648] HybridClock initialized: now 1790704674385742 us; error 41 us; skew 500 ppm
Sep 29 17:57:54 dist-test-slave-r7ph krb5kdc[10021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704674, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:54.389349 10262 init.cc:377] Logged in from keytab as kudu/127.7.118.2@KRBTEST.COM (short username kudu)
I20260929 17:57:54.390633 10262 webserver.cc:460] Webserver started at http://127.7.118.2:39089/ using document root <none> and password file <none>
I20260929 17:57:54.391296 10262 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:54.391403 10262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:54.391669 10262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:54.394074 10262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/data/instance:
uuid: "ee10a94c652549ca8d2e895647ba3731"
format_stamp: "Formatted at 2026-09-29 17:57:54 on dist-test-slave-r7ph"
server_key: "d684f6433d3db5af9e345fb28df1f4a9"
server_key_iv: "2133c95295ae64d69f37c0995d5162c6"
server_key_version: "encryptionkey@0"
I20260929 17:57:54.394949 10262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance:
uuid: "ee10a94c652549ca8d2e895647ba3731"
format_stamp: "Formatted at 2026-09-29 17:57:54 on dist-test-slave-r7ph"
server_key: "d684f6433d3db5af9e345fb28df1f4a9"
server_key_iv: "2133c95295ae64d69f37c0995d5162c6"
server_key_version: "encryptionkey@0"
I20260929 17:57:54.399502 10262 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 17:57:54.402945 10277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:54.404456 10262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 17:57:54.404668 10262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "ee10a94c652549ca8d2e895647ba3731"
format_stamp: "Formatted at 2026-09-29 17:57:54 on dist-test-slave-r7ph"
server_key: "d684f6433d3db5af9e345fb28df1f4a9"
server_key_iv: "2133c95295ae64d69f37c0995d5162c6"
server_key_version: "encryptionkey@0"
I20260929 17:57:54.404822 10262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:54.423225 10262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:54.426925 10262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:54.427156 10262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:54.427796 10262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:57:54.428782 10262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:57:54.428855 10262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:54.428926 10262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:57:54.428979 10262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:54.438971 10262 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:33023
I20260929 17:57:54.439002 10390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:33023 every 8 connection(s)
I20260929 17:57:54.439952 10262 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 17:57:54.445919 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 10262
I20260929 17:57:54.446081 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance
I20260929 17:57:54.446458 7640 external_mini_cluster.cc:1428] Setting key fcaedc6917179f85b41e7598a7dbde83
Sep 29 17:57:54 dist-test-slave-r7ph krb5kdc[10021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704674, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.118.3@KRBTEST.COM; defaulting to no policy
I20260929 17:57:54.457687 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:54.442015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:50449 (local address 127.7.118.62:37087)
0929 17:57:54.442176 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:54.442180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:54.443154 (+ 974us) server_negotiation.cc:407] Connection header received
0929 17:57:54.444509 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:54.444513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:54.444573 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:54.444711 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:54.446309 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:54.447184 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:54.449844 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:54.450279 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:54.450388 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:54.453685 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:54.453689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:54.453691 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:54.453728 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:54.455299 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:54.455926 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:54.455929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:54.455930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:54.456014 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:54.456388 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:54.456392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:54.456393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:54.456639 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:54.456739 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:54.457365 (+ 626us) server_negotiation.cc:299] Negotiation successful
0929 17:57:54.457518 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 17:57:54.458737 10393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:54.442497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:37087 (local address 127.7.118.2:50449)
0929 17:57:54.442985 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:54.443030 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:54.444287 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:54.444927 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:54.444935 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:54.445320 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:54.445933 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:54.445953 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:54.447934 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:54.447948 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:54.449670 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:54.449687 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:54.450490 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:54.450498 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:54.450780 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:54.451502 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:54.451531 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:54.453467 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:54.455468 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:54.455475 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:54.455495 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:54.455796 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:54.456114 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:54.456118 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:54.456120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:54.456285 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:54.456750 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:54.456759 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:54.457112 (+ 353us) client_negotiation.cc:769] Sending connection context
0929 17:57:54.457326 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 17:57:54.457707 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":121,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.3@KRBTEST.COM" created.
I20260929 17:57:54.460331 10391 heartbeater.cc:347] Connected to a master server at 127.7.118.62:37087
I20260929 17:57:54.460719 10391 heartbeater.cc:464] Registering TS with master...
I20260929 17:57:54.461366 10391 heartbeater.cc:511] Master 127.7.118.62:37087 requested a full tablet report, sending...
I20260929 17:57:54.462703 10066 ts_manager.cc:194] Registered new tserver with Master: ee10a94c652549ca8d2e895647ba3731 (127.7.118.2:33023)
I20260929 17:57:54.463555 10066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.2@KRBTEST.COM'} at 127.7.118.2:50449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
I20260929 17:57:54.478413 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:0
--local_ip_for_outbound_sockets=127.7.118.3
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:37087
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:38367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:54.597848 10396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:54.598215 10396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:54.598320 10396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:54.602636 10396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:54.602761 10396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:54.602862 10396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 17:57:54.608893 10396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:38367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:37087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:54.610147 10396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:54.611282 10396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:54.621985 10401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:54.622008 10404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:54.621990 10402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:54.622300 10396 server_base.cc:1034] running on GCE node
I20260929 17:57:54.622905 10396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:54.623602 10396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:54.624787 10396 hybrid_clock.cc:648] HybridClock initialized: now 1790704674624750 us; error 54 us; skew 500 ppm
Sep 29 17:57:54 dist-test-slave-r7ph krb5kdc[10021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704674, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:54.628055 10396 init.cc:377] Logged in from keytab as kudu/127.7.118.3@KRBTEST.COM (short username kudu)
I20260929 17:57:54.629215 10396 webserver.cc:460] Webserver started at http://127.7.118.3:36289/ using document root <none> and password file <none>
I20260929 17:57:54.629874 10396 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:54.629972 10396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:54.630208 10396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:54.632519 10396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/data/instance:
uuid: "7f9846bc47a54e3aa66ad5ff03d1db99"
format_stamp: "Formatted at 2026-09-29 17:57:54 on dist-test-slave-r7ph"
server_key: "e7b178c04c2395439d0476eb0e8189e5"
server_key_iv: "9ec694b03ea8f71575478e9085d75a94"
server_key_version: "encryptionkey@0"
I20260929 17:57:54.633186 10396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance:
uuid: "7f9846bc47a54e3aa66ad5ff03d1db99"
format_stamp: "Formatted at 2026-09-29 17:57:54 on dist-test-slave-r7ph"
server_key: "e7b178c04c2395439d0476eb0e8189e5"
server_key_iv: "9ec694b03ea8f71575478e9085d75a94"
server_key_version: "encryptionkey@0"
I20260929 17:57:54.637606 10396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260929 17:57:54.640538 10411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:54.642138 10396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:57:54.642352 10396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "7f9846bc47a54e3aa66ad5ff03d1db99"
format_stamp: "Formatted at 2026-09-29 17:57:54 on dist-test-slave-r7ph"
server_key: "e7b178c04c2395439d0476eb0e8189e5"
server_key_iv: "9ec694b03ea8f71575478e9085d75a94"
server_key_version: "encryptionkey@0"
I20260929 17:57:54.642495 10396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:54.664454 10396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:54.668301 10396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:54.668785 10396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:54.669515 10396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:57:54.670516 10396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:57:54.670588 10396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:54.670674 10396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:57:54.670717 10396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:54.680379 10396 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:35693
I20260929 17:57:54.680408 10524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:35693 every 8 connection(s)
I20260929 17:57:54.681385 10396 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 17:57:54.686023 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 10396
I20260929 17:57:54.686192 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance
I20260929 17:57:54.686481 7640 external_mini_cluster.cc:1428] Setting key cd9b52ea6609bf69b72e5cc124aba3cf
Sep 29 17:57:54 dist-test-slave-r7ph krb5kdc[10021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704674, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:57:54.696522 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:54.683392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:37393 (local address 127.7.118.62:37087)
0929 17:57:54.683537 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:54.683541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:54.684498 (+ 957us) server_negotiation.cc:407] Connection header received
0929 17:57:54.685933 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:54.685936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:54.686000 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:54.686089 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:54.687655 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:54.688190 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:54.689541 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:54.689773 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:54.689888 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:54.692611 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:54.692614 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:54.692617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:54.692646 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:54.694186 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:54.694829 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:54.694833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:54.694835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:54.694924 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:54.695315 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:54.695319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:54.695320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:54.695523 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:54.695640 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:54.696220 (+ 580us) server_negotiation.cc:299] Negotiation successful
0929 17:57:54.696354 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 17:57:54.697465 10527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:54.683818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:37087 (local address 127.7.118.3:37393)
0929 17:57:54.684318 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:54.684368 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:54.685688 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:54.686361 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:54.686370 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:54.686848 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:54.687495 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:54.687524 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:54.688327 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:54.688331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:54.689396 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:54.689406 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:54.689934 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:54.689939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:54.690128 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:54.690841 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:54.690858 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:54.692391 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:54.694368 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:54.694375 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:54.694395 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:54.694708 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:54.695022 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:54.695025 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:54.695027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:54.695206 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:54.695640 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:54.695649 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:54.695951 (+ 302us) client_negotiation.cc:769] Sending connection context
0929 17:57:54.696163 (+ 212us) client_negotiation.cc:240] Negotiation successful
0929 17:57:54.696548 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":135,"threads_started":1}
I20260929 17:57:54.699010 10525 heartbeater.cc:347] Connected to a master server at 127.7.118.62:37087
I20260929 17:57:54.699321 10525 heartbeater.cc:464] Registering TS with master...
I20260929 17:57:54.700040 10525 heartbeater.cc:511] Master 127.7.118.62:37087 requested a full tablet report, sending...
I20260929 17:57:54.701305 10066 ts_manager.cc:194] Registered new tserver with Master: 7f9846bc47a54e3aa66ad5ff03d1db99 (127.7.118.3:35693)
I20260929 17:57:54.701913 10066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.3@KRBTEST.COM'} at 127.7.118.3:37393
I20260929 17:57:54.713260 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 17:57:54.727056 10105 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:57:54.725438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41134 (local address 127.7.118.62:37087)
0929 17:57:54.725642 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:54.725647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:54.725735 (+ 88us) server_negotiation.cc:407] Connection header received
0929 17:57:54.725893 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:54.725897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:54.725952 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:54.726044 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:54.726904 (+ 860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41134: BlockingRecv error: recv got EOF from 127.0.0.1:41134 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 17:57:54.728221 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 10128
I20260929 17:57:54.734195 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 10262
I20260929 17:57:54.739610 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 10396
I20260929 17:57:54.745235 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 10034
2026-09-29T17:57:54Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2365 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:54 dist-test-slave-r7ph krb5kdc[10537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:54 dist-test-slave-r7ph krb5kdc[10537](info): set up 2 sockets
Sep 29 17:57:54 dist-test-slave-r7ph krb5kdc[10537](info): commencing operation
krb5kdc: starting...
W20260929 17:57:55.850322 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 17:57:55 dist-test-slave-r7ph krb5kdc[10537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 17:57:55.991434 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:35607
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:35607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:56.109624 10550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:56.109998 10550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:56.110078 10550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:56.114497 10550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:56.114626 10550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:56.114661 10550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:56.114701 10550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:56.114735 10550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:56.120715 10550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:35607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:35607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:56.121963 10550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:56.123181 10550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:56.132680 10558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:56.132692 10556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:56.132704 10555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:56.133073 10550 server_base.cc:1034] running on GCE node
I20260929 17:57:56.133855 10550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:56.135241 10550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:56.136442 10550 hybrid_clock.cc:648] HybridClock initialized: now 1790704676136421 us; error 40 us; skew 500 ppm
Sep 29 17:57:56 dist-test-slave-r7ph krb5kdc[10537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704676, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:56.139950 10550 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 17:57:56.141059 10550 webserver.cc:460] Webserver started at http://127.7.118.62:40965/ using document root <none> and password file <none>
I20260929 17:57:56.141724 10550 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:56.141830 10550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:56.142096 10550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:56.144659 10550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "ac8d36687be74133a4ac84be19c8d817"
format_stamp: "Formatted at 2026-09-29 17:57:56 on dist-test-slave-r7ph"
server_key: "12cd010ecd457285105c3de5c24496d9"
server_key_iv: "bd65032c90d85a8e62a40d39ba6fe1e7"
server_key_version: "encryptionkey@0"
I20260929 17:57:56.145380 10550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "ac8d36687be74133a4ac84be19c8d817"
format_stamp: "Formatted at 2026-09-29 17:57:56 on dist-test-slave-r7ph"
server_key: "12cd010ecd457285105c3de5c24496d9"
server_key_iv: "bd65032c90d85a8e62a40d39ba6fe1e7"
server_key_version: "encryptionkey@0"
I20260929 17:57:56.150231 10550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 17:57:56.153324 10565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:56.155102 10550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 17:57:56.155293 10550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "ac8d36687be74133a4ac84be19c8d817"
format_stamp: "Formatted at 2026-09-29 17:57:56 on dist-test-slave-r7ph"
server_key: "12cd010ecd457285105c3de5c24496d9"
server_key_iv: "bd65032c90d85a8e62a40d39ba6fe1e7"
server_key_version: "encryptionkey@0"
I20260929 17:57:56.155447 10550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:56.186900 10550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:56.190814 10550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:56.191067 10550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:56.198671 10550 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:35607
I20260929 17:57:56.198987 10617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:35607 every 8 connection(s)
I20260929 17:57:56.199834 10550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:56.203091 10618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:56.209239 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 10550
I20260929 17:57:56.209295 10618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac8d36687be74133a4ac84be19c8d817: Bootstrap starting.
I20260929 17:57:56.209411 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 17:57:56.209734 7640 external_mini_cluster.cc:1428] Setting key 38e72b24e76f58af3a7617cfe86ebcf3
I20260929 17:57:56.212339 10618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac8d36687be74133a4ac84be19c8d817: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:56.213300 10618 log.cc:824] T 00000000000000000000000000000000 P ac8d36687be74133a4ac84be19c8d817: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:56.215889 10618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac8d36687be74133a4ac84be19c8d817: No bootstrap required, opened a new log
Sep 29 17:57:56 dist-test-slave-r7ph krb5kdc[10537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:57:56.221024 10618 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac8d36687be74133a4ac84be19c8d817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac8d36687be74133a4ac84be19c8d817" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35607 } }
I20260929 17:57:56.221359 10618 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac8d36687be74133a4ac84be19c8d817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:56.221428 10618 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac8d36687be74133a4ac84be19c8d817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac8d36687be74133a4ac84be19c8d817, State: Initialized, Role: FOLLOWER
I20260929 17:57:56.221990 10618 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac8d36687be74133a4ac84be19c8d817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac8d36687be74133a4ac84be19c8d817" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35607 } }
I20260929 17:57:56.222157 10618 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac8d36687be74133a4ac84be19c8d817 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:56.222250 10618 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac8d36687be74133a4ac84be19c8d817 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:56.222397 10618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac8d36687be74133a4ac84be19c8d817 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:56.223698 10618 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac8d36687be74133a4ac84be19c8d817 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac8d36687be74133a4ac84be19c8d817" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35607 } }
I20260929 17:57:56.224105 10618 leader_election.cc:302] T 00000000000000000000000000000000 P ac8d36687be74133a4ac84be19c8d817 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac8d36687be74133a4ac84be19c8d817; no voters:
I20260929 17:57:56.224512 10618 leader_election.cc:288] T 00000000000000000000000000000000 P ac8d36687be74133a4ac84be19c8d817 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:56.224632 10623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac8d36687be74133a4ac84be19c8d817 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:56.224889 10623 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac8d36687be74133a4ac84be19c8d817 [term 1 LEADER]: Becoming Leader. State: Replica: ac8d36687be74133a4ac84be19c8d817, State: Running, Role: LEADER
I20260929 17:57:56.225308 10623 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac8d36687be74133a4ac84be19c8d817 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac8d36687be74133a4ac84be19c8d817" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35607 } }
I20260929 17:57:56.226022 10618 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac8d36687be74133a4ac84be19c8d817 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:56.226627 10621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:56.211121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35620 (local address 127.7.118.62:35607)
0929 17:57:56.211563 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:56.211582 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:56.211630 (+ 48us) server_negotiation.cc:407] Connection header received
0929 17:57:56.212337 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:56.212356 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:56.212794 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:56.213121 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:56.214130 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:56.215298 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:56.216564 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:56.217121 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:56.217263 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:56.220393 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:56.220409 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:56.220428 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:56.220465 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:56.222664 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:56.223231 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:56.223236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:56.223242 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:56.223378 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:56.223715 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:56.223718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:56.223719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:56.224215 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:56.224416 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:56.225194 (+ 778us) server_negotiation.cc:299] Negotiation successful
0929 17:57:56.225524 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":139,"threads_started":1}
I20260929 17:57:56.227172 10625 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac8d36687be74133a4ac84be19c8d817 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac8d36687be74133a4ac84be19c8d817. Latest consensus state: current_term: 1 leader_uuid: "ac8d36687be74133a4ac84be19c8d817" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac8d36687be74133a4ac84be19c8d817" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35607 } } }
I20260929 17:57:56.227310 10625 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac8d36687be74133a4ac84be19c8d817 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:56.227739 10632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:56.228283 10624 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac8d36687be74133a4ac84be19c8d817 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac8d36687be74133a4ac84be19c8d817" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac8d36687be74133a4ac84be19c8d817" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35607 } } }
I20260929 17:57:56.228379 10624 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac8d36687be74133a4ac84be19c8d817 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:56.230569 7640 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:57:56.231518 10632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:56.238408 10632 catalog_manager.cc:1347] Generated new cluster ID: 0d85aec250294645a0b3edd427d14bd6
I20260929 17:57:56.238497 10632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:56.253274 10632 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:56.254645 10632 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:56.258075 10621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:56.252449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35622 (local address 127.7.118.62:35607)
0929 17:57:56.252564 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:56.252569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:56.252995 (+ 426us) server_negotiation.cc:407] Connection header received
0929 17:57:56.253192 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:56.253195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:56.253256 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:56.253350 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:56.254087 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:56.254700 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:56.255659 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:56.255913 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:56.256039 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:56.257022 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:56.257026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:56.257029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:56.257523 (+ 494us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:56.257557 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:56.257574 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:56.257683 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:56.257807 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 17:57:56.257904 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 17:57:56.270371 10632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac8d36687be74133a4ac84be19c8d817: Generated new TSK 0
I20260929 17:57:56.271322 10632 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:57:56.280694 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 10550
2026-09-29T17:57:56Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1531 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:56 dist-test-slave-r7ph krb5kdc[10651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:56 dist-test-slave-r7ph krb5kdc[10651](info): set up 2 sockets
Sep 29 17:57:56 dist-test-slave-r7ph krb5kdc[10651](info): commencing operation
krb5kdc: starting...
W20260929 17:57:57.362418 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 17:57:57 dist-test-slave-r7ph krb5kdc[10651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 17:57:57.497342 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:44725
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:44355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:44725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:57.623282 10664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:57.623615 10664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:57.623662 10664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:57.628070 10664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:57.628166 10664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:57.628188 10664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:57.628206 10664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:57.628221 10664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:57.634421 10664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:44355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:44725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:44725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:57.635692 10664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:57.636854 10664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:57.646682 10669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:57.646689 10670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:57.646806 10664 server_base.cc:1034] running on GCE node
W20260929 17:57:57.646685 10672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:57.647709 10664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:57.649190 10664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:57.650391 10664 hybrid_clock.cc:648] HybridClock initialized: now 1790704677650365 us; error 56 us; skew 500 ppm
Sep 29 17:57:57 dist-test-slave-r7ph krb5kdc[10651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704677, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:57.660622 10664 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 17:57:57.661748 10664 webserver.cc:460] Webserver started at http://127.7.118.62:37759/ using document root <none> and password file <none>
I20260929 17:57:57.662501 10664 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:57.662608 10664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:57.662859 10664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:57.665194 10664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "1b7689d824b4432c9c468f3d48642522"
format_stamp: "Formatted at 2026-09-29 17:57:57 on dist-test-slave-r7ph"
server_key: "0cec927d2fbe02f15ad8cac436e4d0f4"
server_key_iv: "f0ef48ed4946b55a283b12dd9b996d40"
server_key_version: "encryptionkey@0"
I20260929 17:57:57.665831 10664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "1b7689d824b4432c9c468f3d48642522"
format_stamp: "Formatted at 2026-09-29 17:57:57 on dist-test-slave-r7ph"
server_key: "0cec927d2fbe02f15ad8cac436e4d0f4"
server_key_iv: "f0ef48ed4946b55a283b12dd9b996d40"
server_key_version: "encryptionkey@0"
I20260929 17:57:57.670053 10664 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 17:57:57.672989 10679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:57.674527 10664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:57:57.674683 10664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "1b7689d824b4432c9c468f3d48642522"
format_stamp: "Formatted at 2026-09-29 17:57:57 on dist-test-slave-r7ph"
server_key: "0cec927d2fbe02f15ad8cac436e4d0f4"
server_key_iv: "f0ef48ed4946b55a283b12dd9b996d40"
server_key_version: "encryptionkey@0"
I20260929 17:57:57.674817 10664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:57.689908 10664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:57.693656 10664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:57.693893 10664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:57.701121 10664 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:44725
I20260929 17:57:57.701438 10731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:44725 every 8 connection(s)
I20260929 17:57:57.702167 10664 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:57.704202 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 10664
I20260929 17:57:57.704339 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 17:57:57.704612 7640 external_mini_cluster.cc:1428] Setting key 26c6b857059428db70f2e0ee1ccefade
I20260929 17:57:57.705355 10732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:57.712656 10732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b7689d824b4432c9c468f3d48642522: Bootstrap starting.
Sep 29 17:57:57 dist-test-slave-r7ph krb5kdc[10651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:57:57.715474 10732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b7689d824b4432c9c468f3d48642522: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:57.716428 10732 log.cc:824] T 00000000000000000000000000000000 P 1b7689d824b4432c9c468f3d48642522: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:57.719149 10732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b7689d824b4432c9c468f3d48642522: No bootstrap required, opened a new log
I20260929 17:57:57.721423 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:57.706030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39944 (local address 127.7.118.62:44725)
0929 17:57:57.706463 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:57.706484 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:57.706527 (+ 43us) server_negotiation.cc:407] Connection header received
0929 17:57:57.707436 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:57.707470 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:57.707921 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:57.708339 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:57.709066 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:57.710609 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:57.711737 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:57.712466 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:57.712637 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:57.715823 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:57.715830 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:57.715849 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:57.715888 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:57.717876 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:57.718483 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:57.718490 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:57.718496 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:57.718635 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:57.718976 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:57.718979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:57.718981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:57.719512 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:57.719709 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:57.720019 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 17:57:57.720381 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":156,"threads_started":1}
I20260929 17:57:57.723699 10732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b7689d824b4432c9c468f3d48642522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7689d824b4432c9c468f3d48642522" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 44725 } }
I20260929 17:57:57.724035 10732 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b7689d824b4432c9c468f3d48642522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:57.724105 10732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b7689d824b4432c9c468f3d48642522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b7689d824b4432c9c468f3d48642522, State: Initialized, Role: FOLLOWER
I20260929 17:57:57.724583 10732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b7689d824b4432c9c468f3d48642522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7689d824b4432c9c468f3d48642522" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 44725 } }
I20260929 17:57:57.724740 10732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b7689d824b4432c9c468f3d48642522 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:57.724805 10732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b7689d824b4432c9c468f3d48642522 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:57.724912 10732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b7689d824b4432c9c468f3d48642522 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:57.725905 10732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b7689d824b4432c9c468f3d48642522 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7689d824b4432c9c468f3d48642522" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 44725 } }
I20260929 17:57:57.726330 10732 leader_election.cc:302] T 00000000000000000000000000000000 P 1b7689d824b4432c9c468f3d48642522 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b7689d824b4432c9c468f3d48642522; no voters:
I20260929 17:57:57.726733 10732 leader_election.cc:288] T 00000000000000000000000000000000 P 1b7689d824b4432c9c468f3d48642522 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:57.726871 10737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b7689d824b4432c9c468f3d48642522 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:57.727167 10737 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b7689d824b4432c9c468f3d48642522 [term 1 LEADER]: Becoming Leader. State: Replica: 1b7689d824b4432c9c468f3d48642522, State: Running, Role: LEADER
I20260929 17:57:57.727579 10737 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b7689d824b4432c9c468f3d48642522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7689d824b4432c9c468f3d48642522" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 44725 } }
I20260929 17:57:57.728176 10732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b7689d824b4432c9c468f3d48642522 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:57.729498 10739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b7689d824b4432c9c468f3d48642522 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b7689d824b4432c9c468f3d48642522. Latest consensus state: current_term: 1 leader_uuid: "1b7689d824b4432c9c468f3d48642522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7689d824b4432c9c468f3d48642522" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 44725 } } }
I20260929 17:57:57.729629 10739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b7689d824b4432c9c468f3d48642522 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:57.729588 10738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b7689d824b4432c9c468f3d48642522 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b7689d824b4432c9c468f3d48642522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7689d824b4432c9c468f3d48642522" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 44725 } } }
I20260929 17:57:57.729727 10738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b7689d824b4432c9c468f3d48642522 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:57.730033 10746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:57.733094 10746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:57.738948 10746 catalog_manager.cc:1347] Generated new cluster ID: 28a977f8013b4cd995595fd79a2723cd
I20260929 17:57:57.739049 10746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:57.751139 10746 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:57.752154 10746 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:57.759265 10746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b7689d824b4432c9c468f3d48642522: Generated new TSK 0
I20260929 17:57:57.760164 10746 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.118.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
I20260929 17:57:57.806356 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:0
--local_ip_for_outbound_sockets=127.7.118.1
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:44725
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:44355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:57.928748 10758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:57.929047 10758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:57.929098 10758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:57.933538 10758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:57.933614 10758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:57.933677 10758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 17:57:57.939491 10758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:44355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:44725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:57.940789 10758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:57.942163 10758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:57.952673 10763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:57.952626 10764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:57.952668 10766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:57.952858 10758 server_base.cc:1034] running on GCE node
I20260929 17:57:57.953439 10758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:57.954303 10758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:57.955504 10758 hybrid_clock.cc:648] HybridClock initialized: now 1790704677955478 us; error 38 us; skew 500 ppm
Sep 29 17:57:57 dist-test-slave-r7ph krb5kdc[10651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704677, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:57.958478 10758 init.cc:377] Logged in from keytab as kudu/127.7.118.1@KRBTEST.COM (short username kudu)
I20260929 17:57:57.959419 10758 webserver.cc:460] Webserver started at http://127.7.118.1:37039/ using document root <none> and password file <none>
I20260929 17:57:57.959971 10758 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:57.960042 10758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:57.960212 10758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:57.962801 10758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/data/instance:
uuid: "076ba91cb9ff43b6b54bcafd8a46cc1b"
format_stamp: "Formatted at 2026-09-29 17:57:57 on dist-test-slave-r7ph"
server_key: "1e2bddba6a463ee60272d25973504ac6"
server_key_iv: "7d6dad63b30826690f810226fba7f333"
server_key_version: "encryptionkey@0"
I20260929 17:57:57.963536 10758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance:
uuid: "076ba91cb9ff43b6b54bcafd8a46cc1b"
format_stamp: "Formatted at 2026-09-29 17:57:57 on dist-test-slave-r7ph"
server_key: "1e2bddba6a463ee60272d25973504ac6"
server_key_iv: "7d6dad63b30826690f810226fba7f333"
server_key_version: "encryptionkey@0"
I20260929 17:57:57.968201 10758 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:57:57.971130 10773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:57.972714 10758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:57:57.972918 10758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "076ba91cb9ff43b6b54bcafd8a46cc1b"
format_stamp: "Formatted at 2026-09-29 17:57:57 on dist-test-slave-r7ph"
server_key: "1e2bddba6a463ee60272d25973504ac6"
server_key_iv: "7d6dad63b30826690f810226fba7f333"
server_key_version: "encryptionkey@0"
I20260929 17:57:57.973138 10758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:57.995404 10758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:57.999564 10758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:57.999823 10758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:58.000506 10758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:57:58.001603 10758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:57:58.001683 10758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:58.001765 10758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:57:58.001818 10758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:58.012599 10758 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:37877
I20260929 17:57:58.012640 10886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:37877 every 8 connection(s)
I20260929 17:57:58.013728 10758 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 17:57:58.014106 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 10758
I20260929 17:57:58.014293 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance
I20260929 17:57:58.014626 7640 external_mini_cluster.cc:1428] Setting key 3401f790406c14cc2858f873597a60ec
WARNING: no policy specified for kudu/127.7.118.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.2@KRBTEST.COM" created.
Sep 29 17:57:58 dist-test-slave-r7ph krb5kdc[10651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704677, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:57:58.033816 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:58.016135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:47393 (local address 127.7.118.62:44725)
0929 17:57:58.016285 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:58.016290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:58.017639 (+ 1349us) server_negotiation.cc:407] Connection header received
0929 17:57:58.019196 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:58.019199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:58.019258 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:58.019347 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:58.020963 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:58.021955 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:58.023364 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:58.023814 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:58.023940 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:58.029228 (+ 5288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:58.029236 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:58.029242 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:58.029286 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:58.031365 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:58.032035 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:58.032038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:58.032039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:58.032128 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:58.032581 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:58.032584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:58.032586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:58.032794 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:58.032907 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:58.033485 (+ 578us) server_negotiation.cc:299] Negotiation successful
0929 17:57:58.033640 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 17:57:58.034813 10889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:58.016815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:44725 (local address 127.7.118.1:47393)
0929 17:57:58.017434 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:58.017482 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:58.018943 (+ 1461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:58.019659 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:58.019674 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:58.020096 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:58.020766 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:58.020786 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:58.022137 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:58.022140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:58.023210 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:58.023222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:58.024015 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:58.024019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:58.024202 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:58.024893 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:58.024911 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:58.028945 (+ 4034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:58.031530 (+ 2585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:58.031537 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:58.031559 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:58.031889 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:58.032247 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:58.032250 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:58.032252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:58.032426 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:58.032908 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:58.032919 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:58.033219 (+ 300us) client_negotiation.cc:769] Sending connection context
0929 17:57:58.033428 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 17:57:58.033830 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":178,"threads_started":1}
I20260929 17:57:58.036314 10887 heartbeater.cc:347] Connected to a master server at 127.7.118.62:44725
I20260929 17:57:58.036598 10887 heartbeater.cc:464] Registering TS with master...
I20260929 17:57:58.037217 10887 heartbeater.cc:511] Master 127.7.118.62:44725 requested a full tablet report, sending...
I20260929 17:57:58.039110 10696 ts_manager.cc:194] Registered new tserver with Master: 076ba91cb9ff43b6b54bcafd8a46cc1b (127.7.118.1:37877)
I20260929 17:57:58.040648 10696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.1@KRBTEST.COM'} at 127.7.118.1:47393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
I20260929 17:57:58.045854 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:0
--local_ip_for_outbound_sockets=127.7.118.2
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:44725
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:44355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:58.168711 10892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:58.169031 10892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:58.169080 10892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:58.174350 10892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:58.174520 10892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:58.174736 10892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 17:57:58.181217 10892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:44355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:44725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:58.182893 10892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:58.184074 10892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:58.195783 10897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:58.195783 10898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:58.195992 10900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:58.195835 10892 server_base.cc:1034] running on GCE node
I20260929 17:57:58.196772 10892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:58.197584 10892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:58.198791 10892 hybrid_clock.cc:648] HybridClock initialized: now 1790704678198771 us; error 38 us; skew 500 ppm
Sep 29 17:57:58 dist-test-slave-r7ph krb5kdc[10651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704678, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:58.202003 10892 init.cc:377] Logged in from keytab as kudu/127.7.118.2@KRBTEST.COM (short username kudu)
I20260929 17:57:58.203079 10892 webserver.cc:460] Webserver started at http://127.7.118.2:38185/ using document root <none> and password file <none>
I20260929 17:57:58.203675 10892 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:58.203738 10892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:58.203927 10892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:58.206501 10892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/data/instance:
uuid: "111a972328ec48ce97208092c091e2ad"
format_stamp: "Formatted at 2026-09-29 17:57:58 on dist-test-slave-r7ph"
server_key: "e0cbb8bc1b9ae860bd60cfea418e2856"
server_key_iv: "1be097b7ece33ecbae8783b5106abd05"
server_key_version: "encryptionkey@0"
I20260929 17:57:58.207135 10892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance:
uuid: "111a972328ec48ce97208092c091e2ad"
format_stamp: "Formatted at 2026-09-29 17:57:58 on dist-test-slave-r7ph"
server_key: "e0cbb8bc1b9ae860bd60cfea418e2856"
server_key_iv: "1be097b7ece33ecbae8783b5106abd05"
server_key_version: "encryptionkey@0"
I20260929 17:57:58.212260 10892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 17:57:58.215613 10907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:58.217268 10892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:57:58.217412 10892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "111a972328ec48ce97208092c091e2ad"
format_stamp: "Formatted at 2026-09-29 17:57:58 on dist-test-slave-r7ph"
server_key: "e0cbb8bc1b9ae860bd60cfea418e2856"
server_key_iv: "1be097b7ece33ecbae8783b5106abd05"
server_key_version: "encryptionkey@0"
I20260929 17:57:58.217618 10892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:58.231762 10892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:58.235767 10892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:58.235970 10892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:58.236591 10892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:57:58.237498 10892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:57:58.237552 10892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:58.237594 10892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:57:58.237610 10892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:58.247394 10892 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:33615
I20260929 17:57:58.247414 11020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:33615 every 8 connection(s)
I20260929 17:57:58.248387 10892 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 17:57:58.253542 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 10892
I20260929 17:57:58.253647 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance
I20260929 17:57:58.253948 7640 external_mini_cluster.cc:1428] Setting key cae1929631b0c24a974ae5c06ba4027c
Sep 29 17:57:58 dist-test-slave-r7ph krb5kdc[10651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704678, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.118.3@KRBTEST.COM; defaulting to no policy
I20260929 17:57:58.265115 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:58.250451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:33589 (local address 127.7.118.62:44725)
0929 17:57:58.250608 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:58.250614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:58.251589 (+ 975us) server_negotiation.cc:407] Connection header received
0929 17:57:58.253019 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:58.253022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:58.253082 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:58.253175 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:58.255001 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:58.255700 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:58.257081 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:58.257458 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:58.257603 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:58.260935 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:58.260940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:58.260943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:58.260978 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:58.262686 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:58.263335 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:58.263341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:58.263343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:58.263430 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:58.263832 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:58.263835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:58.263837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:58.264038 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:58.264153 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:58.264736 (+ 583us) server_negotiation.cc:299] Negotiation successful
0929 17:57:58.264944 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 17:57:58.266175 11023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:58.250875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:44725 (local address 127.7.118.2:33589)
0929 17:57:58.251378 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:58.251424 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:58.252756 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:58.253470 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:58.253483 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:58.254039 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:58.254780 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:58.254814 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:58.255889 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:58.255892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:58.256896 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:58.256908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:58.257675 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:58.257679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:58.257873 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:58.258622 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:58.258640 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:58.260708 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:58.262851 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:58.262859 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:58.262880 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:58.263211 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:58.263538 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:58.263541 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:58.263543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:58.263722 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:58.264170 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:58.264178 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:58.264474 (+ 296us) client_negotiation.cc:769] Sending connection context
0929 17:57:58.264686 (+ 212us) client_negotiation.cc:240] Negotiation successful
0929 17:57:58.265101 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":128,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.3@KRBTEST.COM" created.
I20260929 17:57:58.267795 11021 heartbeater.cc:347] Connected to a master server at 127.7.118.62:44725
I20260929 17:57:58.268078 11021 heartbeater.cc:464] Registering TS with master...
I20260929 17:57:58.268625 11021 heartbeater.cc:511] Master 127.7.118.62:44725 requested a full tablet report, sending...
I20260929 17:57:58.269842 10696 ts_manager.cc:194] Registered new tserver with Master: 111a972328ec48ce97208092c091e2ad (127.7.118.2:33615)
I20260929 17:57:58.270771 10696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.2@KRBTEST.COM'} at 127.7.118.2:33589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
I20260929 17:57:58.286815 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:0
--local_ip_for_outbound_sockets=127.7.118.3
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:44725
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:44355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:58.412863 11026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:58.413173 11026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:58.413221 11026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:58.417542 11026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:58.417634 11026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:58.417696 11026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 17:57:58.423492 11026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:44355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:44725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:58.424629 11026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:58.425652 11026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:58.436568 11034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:58.436568 11032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:58.436717 11031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:58.437065 11026 server_base.cc:1034] running on GCE node
I20260929 17:57:58.437590 11026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:58.438383 11026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:58.439586 11026 hybrid_clock.cc:648] HybridClock initialized: now 1790704678439568 us; error 31 us; skew 500 ppm
Sep 29 17:57:58 dist-test-slave-r7ph krb5kdc[10651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704678, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:58.442766 11026 init.cc:377] Logged in from keytab as kudu/127.7.118.3@KRBTEST.COM (short username kudu)
I20260929 17:57:58.443861 11026 webserver.cc:460] Webserver started at http://127.7.118.3:42309/ using document root <none> and password file <none>
I20260929 17:57:58.444720 11026 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:58.444803 11026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:58.445008 11026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:58.447589 11026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/data/instance:
uuid: "6bd593eca8cd413aa8707b962dc6e77c"
format_stamp: "Formatted at 2026-09-29 17:57:58 on dist-test-slave-r7ph"
server_key: "e466643e1e4b809fdef96a2847ee423c"
server_key_iv: "86449f80d91a92638a3f95de5565b98e"
server_key_version: "encryptionkey@0"
I20260929 17:57:58.448181 11026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance:
uuid: "6bd593eca8cd413aa8707b962dc6e77c"
format_stamp: "Formatted at 2026-09-29 17:57:58 on dist-test-slave-r7ph"
server_key: "e466643e1e4b809fdef96a2847ee423c"
server_key_iv: "86449f80d91a92638a3f95de5565b98e"
server_key_version: "encryptionkey@0"
I20260929 17:57:58.452605 11026 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:57:58.455593 11041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:58.457144 11026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:57:58.457314 11026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "6bd593eca8cd413aa8707b962dc6e77c"
format_stamp: "Formatted at 2026-09-29 17:57:58 on dist-test-slave-r7ph"
server_key: "e466643e1e4b809fdef96a2847ee423c"
server_key_iv: "86449f80d91a92638a3f95de5565b98e"
server_key_version: "encryptionkey@0"
I20260929 17:57:58.457454 11026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:58.477499 11026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:58.481470 11026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:58.481715 11026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:58.482462 11026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:57:58.483353 11026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:57:58.483402 11026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:58.483472 11026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:57:58.483505 11026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:58.492911 11026 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:46289
I20260929 17:57:58.492949 11154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:46289 every 8 connection(s)
I20260929 17:57:58.493969 11026 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 17:57:58.494302 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 11026
I20260929 17:57:58.494436 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance
I20260929 17:57:58.494756 7640 external_mini_cluster.cc:1428] Setting key ce4c4e143461aab5f4d340026dc46816
Sep 29 17:57:58 dist-test-slave-r7ph krb5kdc[10651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704678, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:57:58.509434 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:58.496060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:35101 (local address 127.7.118.62:44725)
0929 17:57:58.496206 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:58.496212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:58.497369 (+ 1157us) server_negotiation.cc:407] Connection header received
0929 17:57:58.498845 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:58.498848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:58.498902 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:58.498982 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:58.500287 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:58.500833 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:58.502027 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:58.502306 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:58.502433 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:58.505037 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:58.505041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:58.505043 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:58.505069 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:58.506616 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:58.507247 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:58.507250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:58.507251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:58.507329 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:58.507806 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:58.507809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:58.507811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:58.508190 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:58.508301 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:58.509039 (+ 738us) server_negotiation.cc:299] Negotiation successful
0929 17:57:58.509209 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 17:57:58.510366 11157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:58.496642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:44725 (local address 127.7.118.3:35101)
0929 17:57:58.497225 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:58.497271 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:58.498632 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:58.499187 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:58.499199 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:58.499604 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:58.500119 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:58.500133 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:58.500972 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:58.500976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:58.501887 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:58.501896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:58.502467 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:58.502471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:58.502657 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:58.503305 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:58.503322 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:58.504834 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:58.506757 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:58.506764 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:58.506790 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:58.507111 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:58.507424 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:58.507428 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:58.507430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:58.507637 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:58.508321 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:58.508330 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:58.508708 (+ 378us) client_negotiation.cc:769] Sending connection context
0929 17:57:58.508981 (+ 273us) client_negotiation.cc:240] Negotiation successful
0929 17:57:58.509388 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":153,"threads_started":1}
I20260929 17:57:58.511592 11155 heartbeater.cc:347] Connected to a master server at 127.7.118.62:44725
I20260929 17:57:58.511901 11155 heartbeater.cc:464] Registering TS with master...
I20260929 17:57:58.512498 11155 heartbeater.cc:511] Master 127.7.118.62:44725 requested a full tablet report, sending...
I20260929 17:57:58.513619 10696 ts_manager.cc:194] Registered new tserver with Master: 6bd593eca8cd413aa8707b962dc6e77c (127.7.118.3:46289)
I20260929 17:57:58.514216 10696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.3@KRBTEST.COM'} at 127.7.118.3:35101
I20260929 17:57:58.521682 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 17:57:58 dist-test-slave-r7ph krb5kdc[10651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704678, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 17:57:58 dist-test-slave-r7ph krb5kdc[10651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704678, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:57:58.549690 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:58.540290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39956 (local address 127.7.118.62:44725)
0929 17:57:58.540515 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:58.540519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:58.540657 (+ 138us) server_negotiation.cc:407] Connection header received
0929 17:57:58.540838 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:58.540841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:58.540894 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:58.540983 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:58.541735 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:58.542357 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:58.543211 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:58.543445 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:58.543566 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:58.546371 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:58.546376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:58.546378 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:58.546407 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:58.547817 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:58.548277 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:58.548281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:58.548283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:58.548379 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:58.548650 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:58.548655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:58.548657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:58.548909 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:58.549077 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:58.549315 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 17:57:58.549519 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
W20260929 17:57:58.550151 10696 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39956
I20260929 17:57:58.551208 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 10758
I20260929 17:57:58.556870 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 10892
I20260929 17:57:58.562662 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 11026
I20260929 17:57:58.568248 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 10664
2026-09-29T17:57:58Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2291 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:58 dist-test-slave-r7ph krb5kdc[11167](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:58 dist-test-slave-r7ph krb5kdc[11167](info): set up 2 sockets
Sep 29 17:57:58 dist-test-slave-r7ph krb5kdc[11167](info): commencing operation
krb5kdc: starting...
W20260929 17:57:59.662540 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 17:57:59 dist-test-slave-r7ph krb5kdc[11167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 17:57:59.803074 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:44831
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:44831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:59.931125 11180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:59.931440 11180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:59.931483 11180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:59.936583 11180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:59.936682 11180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:59.936703 11180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:59.936721 11180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:59.936736 11180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:59.943404 11180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:44831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:44831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:59.944671 11180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:59.945801 11180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:57:59.956401 11180 server_base.cc:1034] running on GCE node
W20260929 17:57:59.956346 11186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:59.956321 11185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:59.956321 11188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:59.957350 11180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:59.958768 11180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:59.960024 11180 hybrid_clock.cc:648] HybridClock initialized: now 1790704679959983 us; error 57 us; skew 500 ppm
Sep 29 17:57:59 dist-test-slave-r7ph krb5kdc[11167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704679, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:59.963963 11180 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 17:57:59.965047 11180 webserver.cc:460] Webserver started at http://127.7.118.62:36545/ using document root <none> and password file <none>
I20260929 17:57:59.965693 11180 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:59.965786 11180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:59.966032 11180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:59.968493 11180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "e80de76dc5204b92b6442da848a1b9b9"
format_stamp: "Formatted at 2026-09-29 17:57:59 on dist-test-slave-r7ph"
server_key: "89d0263779861afae74747db54c283fb"
server_key_iv: "4df01e2b8a1722207dcb6ce209472712"
server_key_version: "encryptionkey@0"
I20260929 17:57:59.969169 11180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "e80de76dc5204b92b6442da848a1b9b9"
format_stamp: "Formatted at 2026-09-29 17:57:59 on dist-test-slave-r7ph"
server_key: "89d0263779861afae74747db54c283fb"
server_key_iv: "4df01e2b8a1722207dcb6ce209472712"
server_key_version: "encryptionkey@0"
I20260929 17:57:59.973608 11180 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260929 17:57:59.976869 11195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:59.978859 11180 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 17:57:59.979076 11180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "e80de76dc5204b92b6442da848a1b9b9"
format_stamp: "Formatted at 2026-09-29 17:57:59 on dist-test-slave-r7ph"
server_key: "89d0263779861afae74747db54c283fb"
server_key_iv: "4df01e2b8a1722207dcb6ce209472712"
server_key_version: "encryptionkey@0"
I20260929 17:57:59.979244 11180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:59.995765 11180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:00.000773 11180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:00.001034 11180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:00.009203 11180 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:44831
I20260929 17:58:00.009219 11247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:44831 every 8 connection(s)
I20260929 17:58:00.010407 11180 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 17:58:00.012632 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 11180
I20260929 17:58:00.012960 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 17:58:00.013325 7640 external_mini_cluster.cc:1428] Setting key a3fa0c1d53ac30d0cd6d6df17ee8a9d1
I20260929 17:58:00.014772 11248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:00.023634 11248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e80de76dc5204b92b6442da848a1b9b9: Bootstrap starting.
Sep 29 17:58:00 dist-test-slave-r7ph krb5kdc[11167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:58:00.027070 11248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e80de76dc5204b92b6442da848a1b9b9: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:00.028221 11248 log.cc:824] T 00000000000000000000000000000000 P e80de76dc5204b92b6442da848a1b9b9: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:00.030689 11248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e80de76dc5204b92b6442da848a1b9b9: No bootstrap required, opened a new log
I20260929 17:58:00.033650 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:00.015432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54512 (local address 127.7.118.62:44831)
0929 17:58:00.016311 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:00.016348 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:00.016396 (+ 48us) server_negotiation.cc:407] Connection header received
0929 17:58:00.017194 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:00.017216 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:00.017623 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:00.017968 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:00.019093 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:00.020279 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:00.021902 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:00.022612 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:00.022769 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:00.026301 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:00.026315 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:00.026334 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:00.026374 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:00.029450 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:00.030151 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:00.030157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:00.030162 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:00.030377 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:00.031298 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:00.031302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:00.031304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:00.031846 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:00.032085 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:00.032463 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 17:58:00.032809 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":178,"threads_started":1}
I20260929 17:58:00.035215 11248 raft_consensus.cc:348] T 00000000000000000000000000000000 P e80de76dc5204b92b6442da848a1b9b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80de76dc5204b92b6442da848a1b9b9" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 44831 } }
I20260929 17:58:00.035586 11248 raft_consensus.cc:374] T 00000000000000000000000000000000 P e80de76dc5204b92b6442da848a1b9b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:00.035676 11248 raft_consensus.cc:729] T 00000000000000000000000000000000 P e80de76dc5204b92b6442da848a1b9b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e80de76dc5204b92b6442da848a1b9b9, State: Initialized, Role: FOLLOWER
I20260929 17:58:00.036370 11248 consensus_queue.cc:260] T 00000000000000000000000000000000 P e80de76dc5204b92b6442da848a1b9b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80de76dc5204b92b6442da848a1b9b9" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 44831 } }
I20260929 17:58:00.036586 11248 raft_consensus.cc:388] T 00000000000000000000000000000000 P e80de76dc5204b92b6442da848a1b9b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:58:00.036743 11248 raft_consensus.cc:482] T 00000000000000000000000000000000 P e80de76dc5204b92b6442da848a1b9b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:58:00.036868 11248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e80de76dc5204b92b6442da848a1b9b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:00.037901 11248 raft_consensus.cc:504] T 00000000000000000000000000000000 P e80de76dc5204b92b6442da848a1b9b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80de76dc5204b92b6442da848a1b9b9" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 44831 } }
I20260929 17:58:00.038338 11248 leader_election.cc:302] T 00000000000000000000000000000000 P e80de76dc5204b92b6442da848a1b9b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e80de76dc5204b92b6442da848a1b9b9; no voters:
I20260929 17:58:00.038697 11248 leader_election.cc:288] T 00000000000000000000000000000000 P e80de76dc5204b92b6442da848a1b9b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:58:00.038866 11253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e80de76dc5204b92b6442da848a1b9b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:58:00.039170 11253 raft_consensus.cc:686] T 00000000000000000000000000000000 P e80de76dc5204b92b6442da848a1b9b9 [term 1 LEADER]: Becoming Leader. State: Replica: e80de76dc5204b92b6442da848a1b9b9, State: Running, Role: LEADER
I20260929 17:58:00.039547 11253 consensus_queue.cc:237] T 00000000000000000000000000000000 P e80de76dc5204b92b6442da848a1b9b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80de76dc5204b92b6442da848a1b9b9" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 44831 } }
I20260929 17:58:00.040103 11248 sys_catalog.cc:564] T 00000000000000000000000000000000 P e80de76dc5204b92b6442da848a1b9b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:58:00.041822 11254 sys_catalog.cc:455] T 00000000000000000000000000000000 P e80de76dc5204b92b6442da848a1b9b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e80de76dc5204b92b6442da848a1b9b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80de76dc5204b92b6442da848a1b9b9" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 44831 } } }
I20260929 17:58:00.042004 11254 sys_catalog.cc:458] T 00000000000000000000000000000000 P e80de76dc5204b92b6442da848a1b9b9 [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:00.041797 11255 sys_catalog.cc:455] T 00000000000000000000000000000000 P e80de76dc5204b92b6442da848a1b9b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e80de76dc5204b92b6442da848a1b9b9. Latest consensus state: current_term: 1 leader_uuid: "e80de76dc5204b92b6442da848a1b9b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80de76dc5204b92b6442da848a1b9b9" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 44831 } } }
I20260929 17:58:00.042100 11255 sys_catalog.cc:458] T 00000000000000000000000000000000 P e80de76dc5204b92b6442da848a1b9b9 [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:00.042500 11262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:58:00.046855 11262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:58:00.054769 11262 catalog_manager.cc:1347] Generated new cluster ID: 6c232a7c78414f5aab5c7309ad636cd0
I20260929 17:58:00.054860 11262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:58:00.064046 11262 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:58:00.065239 11262 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:58:00.071923 11262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e80de76dc5204b92b6442da848a1b9b9: Generated new TSK 0
I20260929 17:58:00.073166 11262 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.118.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
I20260929 17:58:00.119323 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:0
--local_ip_for_outbound_sockets=127.7.118.1
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:44831
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:00.253191 11274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:00.253561 11274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:00.253652 11274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:00.258735 11274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:00.258863 11274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:00.258971 11274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 17:58:00.265517 11274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:44831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:00.266969 11274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:00.268440 11274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:00.280787 11279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:00.280905 11282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:00.280946 11274 server_base.cc:1034] running on GCE node
W20260929 17:58:00.280812 11280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:00.281802 11274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:00.282581 11274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:00.283812 11274 hybrid_clock.cc:648] HybridClock initialized: now 1790704680283787 us; error 41 us; skew 500 ppm
Sep 29 17:58:00 dist-test-slave-r7ph krb5kdc[11167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704680, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:00.287652 11274 init.cc:377] Logged in from keytab as kudu/127.7.118.1@KRBTEST.COM (short username kudu)
I20260929 17:58:00.288857 11274 webserver.cc:460] Webserver started at http://127.7.118.1:46399/ using document root <none> and password file <none>
I20260929 17:58:00.289505 11274 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:00.289574 11274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:00.289805 11274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:00.292146 11274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-0/data/instance:
uuid: "ba1501eefd414782b89357ed331337e2"
format_stamp: "Formatted at 2026-09-29 17:58:00 on dist-test-slave-r7ph"
server_key: "afe7d07a20fa3118ca3ace0dc8d9bfd2"
server_key_iv: "e5ac212c2fa46635e9618cde771b0f0b"
server_key_version: "encryptionkey@0"
I20260929 17:58:00.292765 11274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance:
uuid: "ba1501eefd414782b89357ed331337e2"
format_stamp: "Formatted at 2026-09-29 17:58:00 on dist-test-slave-r7ph"
server_key: "afe7d07a20fa3118ca3ace0dc8d9bfd2"
server_key_iv: "e5ac212c2fa46635e9618cde771b0f0b"
server_key_version: "encryptionkey@0"
I20260929 17:58:00.297060 11274 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 17:58:00.300196 11289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:00.301843 11274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:58:00.301985 11274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "ba1501eefd414782b89357ed331337e2"
format_stamp: "Formatted at 2026-09-29 17:58:00 on dist-test-slave-r7ph"
server_key: "afe7d07a20fa3118ca3ace0dc8d9bfd2"
server_key_iv: "e5ac212c2fa46635e9618cde771b0f0b"
server_key_version: "encryptionkey@0"
I20260929 17:58:00.302153 11274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:00.314922 11274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:00.318845 11274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:00.319084 11274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:00.319725 11274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:00.320696 11274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:00.320746 11274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:00.320788 11274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:00.320829 11274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:00.330561 11274 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:42925
I20260929 17:58:00.330607 11402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:42925 every 8 connection(s)
I20260929 17:58:00.331585 11274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 17:58:00.337744 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 11274
I20260929 17:58:00.337868 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance
I20260929 17:58:00.338191 7640 external_mini_cluster.cc:1428] Setting key 85cdfa500ad01b32e010e427e2f395f8
Sep 29 17:58:00 dist-test-slave-r7ph krb5kdc[11167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704680, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.118.2@KRBTEST.COM; defaulting to no policy
I20260929 17:58:00.349391 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:00.333836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:47779 (local address 127.7.118.62:44831)
0929 17:58:00.334013 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:00.334018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:00.335089 (+ 1071us) server_negotiation.cc:407] Connection header received
0929 17:58:00.336627 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:00.336631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:00.336693 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:00.336778 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:00.338222 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:00.338861 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:00.340313 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:00.340796 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:00.340936 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:00.344291 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:00.344297 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:00.344303 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:00.344340 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:00.346491 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:00.347185 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:00.347190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:00.347192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:00.347304 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:00.347731 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:00.347735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:00.347737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:00.348036 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:00.348163 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:00.348976 (+ 813us) server_negotiation.cc:299] Negotiation successful
0929 17:58:00.349189 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 17:58:00.350576 11405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:00.334337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:44831 (local address 127.7.118.1:47779)
0929 17:58:00.334889 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:00.334940 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:00.336370 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:00.336998 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:00.337005 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:00.337411 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:00.338055 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:00.338074 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:00.339019 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:00.339022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:00.340161 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:00.340172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:00.341025 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:00.341029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:00.341199 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:00.342032 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:00.342061 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:00.344008 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:00.346670 (+ 2662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:00.346677 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:00.346693 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:00.347060 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:00.347431 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:00.347434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:00.347435 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:00.347615 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:00.348208 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:00.348215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:00.348574 (+ 359us) client_negotiation.cc:769] Sending connection context
0929 17:58:00.348846 (+ 272us) client_negotiation.cc:240] Negotiation successful
0929 17:58:00.349324 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.2@KRBTEST.COM" created.
I20260929 17:58:00.352382 11403 heartbeater.cc:347] Connected to a master server at 127.7.118.62:44831
I20260929 17:58:00.352694 11403 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:00.353299 11403 heartbeater.cc:511] Master 127.7.118.62:44831 requested a full tablet report, sending...
I20260929 17:58:00.355172 11212 ts_manager.cc:194] Registered new tserver with Master: ba1501eefd414782b89357ed331337e2 (127.7.118.1:42925)
I20260929 17:58:00.356464 11212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.1@KRBTEST.COM'} at 127.7.118.1:47779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
I20260929 17:58:00.371294 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:0
--local_ip_for_outbound_sockets=127.7.118.2
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:44831
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:00.495321 11408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:00.495646 11408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:00.495692 11408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:00.500411 11408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:00.500496 11408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:00.500586 11408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 17:58:00.506595 11408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:44831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:00.507913 11408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:00.509019 11408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:00.520203 11414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:00.520203 11413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:00.520304 11416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:00.520714 11408 server_base.cc:1034] running on GCE node
I20260929 17:58:00.521193 11408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:00.521976 11408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:00.523198 11408 hybrid_clock.cc:648] HybridClock initialized: now 1790704680523179 us; error 86 us; skew 500 ppm
Sep 29 17:58:00 dist-test-slave-r7ph krb5kdc[11167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704680, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:00.526835 11408 init.cc:377] Logged in from keytab as kudu/127.7.118.2@KRBTEST.COM (short username kudu)
I20260929 17:58:00.527940 11408 webserver.cc:460] Webserver started at http://127.7.118.2:36661/ using document root <none> and password file <none>
I20260929 17:58:00.528576 11408 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:00.528673 11408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:00.528925 11408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:00.531284 11408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-1/data/instance:
uuid: "d35acaa20c4541fa9d593d406d302c02"
format_stamp: "Formatted at 2026-09-29 17:58:00 on dist-test-slave-r7ph"
server_key: "5c93bb623116e68b2c8ecebb5daf1020"
server_key_iv: "c1474c86ba7fcd4d98b56c9d385a58df"
server_key_version: "encryptionkey@0"
I20260929 17:58:00.531934 11408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance:
uuid: "d35acaa20c4541fa9d593d406d302c02"
format_stamp: "Formatted at 2026-09-29 17:58:00 on dist-test-slave-r7ph"
server_key: "5c93bb623116e68b2c8ecebb5daf1020"
server_key_iv: "c1474c86ba7fcd4d98b56c9d385a58df"
server_key_version: "encryptionkey@0"
I20260929 17:58:00.536305 11408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:58:00.539176 11423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:00.540802 11408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:58:00.540962 11408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "d35acaa20c4541fa9d593d406d302c02"
format_stamp: "Formatted at 2026-09-29 17:58:00 on dist-test-slave-r7ph"
server_key: "5c93bb623116e68b2c8ecebb5daf1020"
server_key_iv: "c1474c86ba7fcd4d98b56c9d385a58df"
server_key_version: "encryptionkey@0"
I20260929 17:58:00.541116 11408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:00.577083 11408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:00.580998 11408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:00.581347 11408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:00.582021 11408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:00.583106 11408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:00.583184 11408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:00.583266 11408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:00.583309 11408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:00.594528 11408 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:35349
I20260929 17:58:00.594551 11536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:35349 every 8 connection(s)
I20260929 17:58:00.595643 11408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 17:58:00.599514 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 11408
I20260929 17:58:00.599620 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance
I20260929 17:58:00.599900 7640 external_mini_cluster.cc:1428] Setting key 76b991481b3ccca106a4e49177853a0a
Sep 29 17:58:00 dist-test-slave-r7ph krb5kdc[11167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704680, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.118.3@KRBTEST.COM; defaulting to no policy
I20260929 17:58:00.614673 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:00.597924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:56009 (local address 127.7.118.62:44831)
0929 17:58:00.598089 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:00.598093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:00.599172 (+ 1079us) server_negotiation.cc:407] Connection header received
0929 17:58:00.601058 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:00.601061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:00.601108 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:00.601190 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:00.602741 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:00.603632 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:00.605720 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:00.606017 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:00.606227 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:00.609809 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:00.609817 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:00.609821 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:00.609863 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:00.611944 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:00.612716 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:00.612720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:00.612721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:00.612815 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:00.613343 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:00.613346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:00.613347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:00.613562 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:00.613650 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:00.614401 (+ 751us) server_negotiation.cc:299] Negotiation successful
0929 17:58:00.614543 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.3@KRBTEST.COM" created.
I20260929 17:58:00.615746 11539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:00.598461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:44831 (local address 127.7.118.2:56009)
0929 17:58:00.598992 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:00.599044 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:00.600757 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:00.601437 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:00.601448 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:00.601918 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:00.602564 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:00.602581 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:00.603819 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:00.603826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:00.605534 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:00.605552 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:00.606296 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:00.606304 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:00.606584 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:00.607509 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:00.607529 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:00.609551 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:00.612136 (+ 2585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:00.612144 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:00.612161 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:00.612558 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:00.612955 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:00.612961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:00.612965 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:00.613218 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:00.613696 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:00.613702 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:00.614007 (+ 305us) client_negotiation.cc:769] Sending connection context
0929 17:58:00.614215 (+ 208us) client_negotiation.cc:240] Negotiation successful
0929 17:58:00.614740 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":119,"threads_started":1}
I20260929 17:58:00.617261 11537 heartbeater.cc:347] Connected to a master server at 127.7.118.62:44831
I20260929 17:58:00.617602 11537 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:00.618199 11537 heartbeater.cc:511] Master 127.7.118.62:44831 requested a full tablet report, sending...
I20260929 17:58:00.619547 11212 ts_manager.cc:194] Registered new tserver with Master: d35acaa20c4541fa9d593d406d302c02 (127.7.118.2:35349)
I20260929 17:58:00.620193 11212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.2@KRBTEST.COM'} at 127.7.118.2:56009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
I20260929 17:58:00.635197 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:0
--local_ip_for_outbound_sockets=127.7.118.3
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:44831
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:00.752343 11542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:00.752696 11542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:00.752780 11542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:00.757261 11542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:00.757372 11542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:00.757467 11542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 17:58:00.763190 11542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:44831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:00.764863 11542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:00.765954 11542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:00.776114 11550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:00.776139 11547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:00.776130 11548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:00.776353 11542 server_base.cc:1034] running on GCE node
I20260929 17:58:00.776923 11542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:00.777609 11542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:00.778785 11542 hybrid_clock.cc:648] HybridClock initialized: now 1790704680778767 us; error 30 us; skew 500 ppm
Sep 29 17:58:00 dist-test-slave-r7ph krb5kdc[11167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704680, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:00.781889 11542 init.cc:377] Logged in from keytab as kudu/127.7.118.3@KRBTEST.COM (short username kudu)
I20260929 17:58:00.782997 11542 webserver.cc:460] Webserver started at http://127.7.118.3:40643/ using document root <none> and password file <none>
I20260929 17:58:00.783684 11542 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:00.783756 11542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:00.783988 11542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:00.786345 11542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-2/data/instance:
uuid: "20a032446044436fa333eeb8202061e8"
format_stamp: "Formatted at 2026-09-29 17:58:00 on dist-test-slave-r7ph"
server_key: "6f4b8a9dd4e3cde865e4c3906548340e"
server_key_iv: "078e916f9abbac67fd89dd7628272a45"
server_key_version: "encryptionkey@0"
I20260929 17:58:00.786959 11542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance:
uuid: "20a032446044436fa333eeb8202061e8"
format_stamp: "Formatted at 2026-09-29 17:58:00 on dist-test-slave-r7ph"
server_key: "6f4b8a9dd4e3cde865e4c3906548340e"
server_key_iv: "078e916f9abbac67fd89dd7628272a45"
server_key_version: "encryptionkey@0"
I20260929 17:58:00.791080 11542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 17:58:00.793932 11557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:00.795532 11542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:58:00.795688 11542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "20a032446044436fa333eeb8202061e8"
format_stamp: "Formatted at 2026-09-29 17:58:00 on dist-test-slave-r7ph"
server_key: "6f4b8a9dd4e3cde865e4c3906548340e"
server_key_iv: "078e916f9abbac67fd89dd7628272a45"
server_key_version: "encryptionkey@0"
I20260929 17:58:00.795823 11542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:00.822970 11542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:00.826655 11542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:00.826895 11542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:00.827601 11542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:00.828577 11542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:00.828629 11542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:00.828707 11542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:00.828745 11542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:00.838958 11542 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:34311
I20260929 17:58:00.838986 11670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:34311 every 8 connection(s)
I20260929 17:58:00.840101 11542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 17:58:00.842897 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 11542
I20260929 17:58:00.843058 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance
I20260929 17:58:00.843412 7640 external_mini_cluster.cc:1428] Setting key 4561a0b7fec9e7c24fcee9ba4f621e24
Sep 29 17:58:00 dist-test-slave-r7ph krb5kdc[11167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704680, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:58:00.856482 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:00.842084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:44301 (local address 127.7.118.62:44831)
0929 17:58:00.842273 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:00.842280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:00.843626 (+ 1346us) server_negotiation.cc:407] Connection header received
0929 17:58:00.845598 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:00.845601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:00.845660 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:00.845771 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:00.847298 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:00.847890 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:00.849299 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:00.849561 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:00.849686 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:00.852541 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:00.852545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:00.852549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:00.852588 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:00.853997 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:00.854696 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:00.854700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:00.854702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:00.854798 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:00.855211 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:00.855214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:00.855216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:00.855435 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:00.855558 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:00.856150 (+ 592us) server_negotiation.cc:299] Negotiation successful
0929 17:58:00.856292 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 17:58:00.857692 11673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:00.842747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:44831 (local address 127.7.118.3:44301)
0929 17:58:00.843415 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:00.843492 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:00.845295 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:00.846043 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:00.846056 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:00.846556 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:00.847125 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:00.847140 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:00.848047 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:00.848051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:00.849131 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:00.849141 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:00.849743 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:00.849748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:00.849949 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:00.850635 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:00.850656 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:00.852321 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:00.854148 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:00.854155 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:00.854179 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:00.854569 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:00.854907 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:00.854911 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:00.854914 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:00.855094 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:00.855569 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:00.855579 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:00.855876 (+ 297us) client_negotiation.cc:769] Sending connection context
0929 17:58:00.856090 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 17:58:00.856494 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":170,"threads_started":1}
I20260929 17:58:00.858958 11671 heartbeater.cc:347] Connected to a master server at 127.7.118.62:44831
I20260929 17:58:00.859227 11671 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:00.859757 11671 heartbeater.cc:511] Master 127.7.118.62:44831 requested a full tablet report, sending...
I20260929 17:58:00.860806 11212 ts_manager.cc:194] Registered new tserver with Master: 20a032446044436fa333eeb8202061e8 (127.7.118.3:34311)
I20260929 17:58:00.861385 11212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.3@KRBTEST.COM'} at 127.7.118.3:44301
I20260929 17:58:00.870157 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 17:58:00 dist-test-slave-r7ph krb5kdc[11167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 17:58:00 dist-test-slave-r7ph krb5kdc[11167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.1@KRBTEST.COM
I20260929 17:58:00.897667 11678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:00.887354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59194 (local address 127.7.118.1:42925)
0929 17:58:00.887718 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:00.887725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:00.887751 (+ 26us) server_negotiation.cc:407] Connection header received
0929 17:58:00.887804 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:00.887809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:00.888019 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:00.888153 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:00.888821 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:00.889713 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:00.890845 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:00.891351 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:00.891489 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:00.893998 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:00.894009 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:00.894025 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:00.894057 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:00.895773 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:00.896225 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:00.896231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:00.896232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:00.896323 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:00.896599 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:00.896601 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:00.896603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:00.896874 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:00.897052 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:00.897252 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 17:58:00.897445 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":144,"threads_started":1}
Sep 29 17:58:00 dist-test-slave-r7ph krb5kdc[11167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:58:00.910187 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:00.901018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54516 (local address 127.7.118.62:44831)
0929 17:58:00.901161 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:00.901169 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:00.901263 (+ 94us) server_negotiation.cc:407] Connection header received
0929 17:58:00.901440 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:00.901443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:00.901503 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:00.901600 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:00.902338 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:00.902924 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:00.903810 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:00.904075 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:00.904200 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:00.906933 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:00.906937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:00.906939 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:00.906965 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:00.908483 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:00.908906 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:00.908909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:00.908910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:00.908983 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:00.909237 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:00.909241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:00.909242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:00.909497 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:00.909611 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:00.909854 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 17:58:00.909966 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 17:58:00.910656 11212 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54516
I20260929 17:58:00.911329 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 11274
I20260929 17:58:00.917330 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 11408
I20260929 17:58:00.922987 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 11542
I20260929 17:58:00.928707 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 11180
2026-09-29T17:58:00Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2359 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:58:00 dist-test-slave-r7ph krb5kdc[11683](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:58:00 dist-test-slave-r7ph krb5kdc[11683](info): set up 2 sockets
Sep 29 17:58:00 dist-test-slave-r7ph krb5kdc[11683](info): commencing operation
krb5kdc: starting...
W20260929 17:58:01.994925 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 17:58:02 dist-test-slave-r7ph krb5kdc[11683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:58:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:58:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 17:58:02.135027 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:44813
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:44813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:02.260362 11696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:02.260715 11696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:02.260805 11696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:02.265230 11696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:58:02.265355 11696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:02.265390 11696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:02.265429 11696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:58:02.265468 11696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:58:02.271319 11696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:44813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:44813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:02.272527 11696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:02.273650 11696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:02.283170 11702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:02.283149 11704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:02.283134 11701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:02.283558 11696 server_base.cc:1034] running on GCE node
I20260929 17:58:02.284372 11696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:02.285681 11696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:02.286906 11696 hybrid_clock.cc:648] HybridClock initialized: now 1790704682286864 us; error 60 us; skew 500 ppm
Sep 29 17:58:02 dist-test-slave-r7ph krb5kdc[11683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704682, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:02.290093 11696 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 17:58:02.291028 11696 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:02.291105 11696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:02.291286 11696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:02.293633 11696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "ee83326e0eee457eb2cdd39e325ec16c"
format_stamp: "Formatted at 2026-09-29 17:58:02 on dist-test-slave-r7ph"
server_key: "c765317352d822777f9880f9f1adf4e2"
server_key_iv: "0a3bdaf914ebbf10b7eb9726c750014e"
server_key_version: "encryptionkey@0"
I20260929 17:58:02.294304 11696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "ee83326e0eee457eb2cdd39e325ec16c"
format_stamp: "Formatted at 2026-09-29 17:58:02 on dist-test-slave-r7ph"
server_key: "c765317352d822777f9880f9f1adf4e2"
server_key_iv: "0a3bdaf914ebbf10b7eb9726c750014e"
server_key_version: "encryptionkey@0"
I20260929 17:58:02.298621 11696 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 17:58:02.301365 11710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:02.303221 11696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:58:02.303390 11696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "ee83326e0eee457eb2cdd39e325ec16c"
format_stamp: "Formatted at 2026-09-29 17:58:02 on dist-test-slave-r7ph"
server_key: "c765317352d822777f9880f9f1adf4e2"
server_key_iv: "0a3bdaf914ebbf10b7eb9726c750014e"
server_key_version: "encryptionkey@0"
I20260929 17:58:02.303510 11696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:02.335565 11696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:02.339535 11696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:02.339726 11696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:02.347916 11696 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:44813
I20260929 17:58:02.348210 11762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:44813 every 8 connection(s)
I20260929 17:58:02.348766 11696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 17:58:02.352035 11763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:02.352764 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 11696
I20260929 17:58:02.352895 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 17:58:02.353220 7640 external_mini_cluster.cc:1428] Setting key ed4f1b5978f2085d55b2aad3db87dec8
I20260929 17:58:02.359004 11763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee83326e0eee457eb2cdd39e325ec16c: Bootstrap starting.
I20260929 17:58:02.361414 11763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee83326e0eee457eb2cdd39e325ec16c: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:02.362215 11763 log.cc:824] T 00000000000000000000000000000000 P ee83326e0eee457eb2cdd39e325ec16c: Log is configured to *not* fsync() on all Append() calls
Sep 29 17:58:02 dist-test-slave-r7ph krb5kdc[11683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:58:02.364835 11763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee83326e0eee457eb2cdd39e325ec16c: No bootstrap required, opened a new log
I20260929 17:58:02.369068 11766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:02.354905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52256 (local address 127.7.118.62:44813)
0929 17:58:02.355408 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:02.355435 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:02.355476 (+ 41us) server_negotiation.cc:407] Connection header received
0929 17:58:02.356468 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:02.356497 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:02.356997 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:02.357360 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:02.358222 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:02.359329 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:02.360222 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:02.360758 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:02.360938 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:02.363850 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:02.363857 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:02.363877 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:02.363923 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:02.365879 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:02.366453 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:02.366460 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:02.366467 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:02.366600 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:02.366908 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:02.366911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:02.366913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:02.367354 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:02.367594 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:02.367900 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 17:58:02.368235 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":156,"threads_started":1}
I20260929 17:58:02.368978 11763 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee83326e0eee457eb2cdd39e325ec16c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee83326e0eee457eb2cdd39e325ec16c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 44813 } }
I20260929 17:58:02.369256 11763 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee83326e0eee457eb2cdd39e325ec16c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:02.369320 11763 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee83326e0eee457eb2cdd39e325ec16c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee83326e0eee457eb2cdd39e325ec16c, State: Initialized, Role: FOLLOWER
I20260929 17:58:02.369840 11763 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee83326e0eee457eb2cdd39e325ec16c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee83326e0eee457eb2cdd39e325ec16c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 44813 } }
I20260929 17:58:02.370028 11763 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee83326e0eee457eb2cdd39e325ec16c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:58:02.370082 11763 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee83326e0eee457eb2cdd39e325ec16c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:58:02.370172 11763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee83326e0eee457eb2cdd39e325ec16c [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:02.371281 11763 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee83326e0eee457eb2cdd39e325ec16c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee83326e0eee457eb2cdd39e325ec16c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 44813 } }
I20260929 17:58:02.371683 11763 leader_election.cc:302] T 00000000000000000000000000000000 P ee83326e0eee457eb2cdd39e325ec16c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee83326e0eee457eb2cdd39e325ec16c; no voters:
I20260929 17:58:02.372094 11763 leader_election.cc:288] T 00000000000000000000000000000000 P ee83326e0eee457eb2cdd39e325ec16c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:58:02.372329 11768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee83326e0eee457eb2cdd39e325ec16c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:58:02.372609 11768 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee83326e0eee457eb2cdd39e325ec16c [term 1 LEADER]: Becoming Leader. State: Replica: ee83326e0eee457eb2cdd39e325ec16c, State: Running, Role: LEADER
I20260929 17:58:02.373046 11768 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee83326e0eee457eb2cdd39e325ec16c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee83326e0eee457eb2cdd39e325ec16c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 44813 } }
I20260929 17:58:02.373584 11763 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee83326e0eee457eb2cdd39e325ec16c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:58:02.375319 11769 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee83326e0eee457eb2cdd39e325ec16c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee83326e0eee457eb2cdd39e325ec16c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee83326e0eee457eb2cdd39e325ec16c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 44813 } } }
I20260929 17:58:02.375514 11769 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee83326e0eee457eb2cdd39e325ec16c [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:02.375967 11770 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee83326e0eee457eb2cdd39e325ec16c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee83326e0eee457eb2cdd39e325ec16c. Latest consensus state: current_term: 1 leader_uuid: "ee83326e0eee457eb2cdd39e325ec16c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee83326e0eee457eb2cdd39e325ec16c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 44813 } } }
I20260929 17:58:02.376786 11770 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee83326e0eee457eb2cdd39e325ec16c [sys.catalog]: This master's current role is: LEADER
W20260929 17:58:02.378643 11783 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee83326e0eee457eb2cdd39e325ec16c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:58:02.378719 11783 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:58:02.378803 11782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:58:02.379818 11782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:58:02.386516 11782 catalog_manager.cc:1347] Generated new cluster ID: 738fdae84f92488fb59e0f225a7dd0d2
I20260929 17:58:02.386616 11782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:58:02.395285 11782 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:58:02.396741 11782 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:58:02.416465 11782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee83326e0eee457eb2cdd39e325ec16c: Generated new TSK 0
I20260929 17:58:02.417497 11782 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.118.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
I20260929 17:58:02.449502 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:0
--local_ip_for_outbound_sockets=127.7.118.1
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:44813
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:02.579562 11789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:02.579864 11789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:02.579916 11789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:02.584167 11789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:02.584252 11789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:02.584323 11789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 17:58:02.589900 11789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:44813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:02.591099 11789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:02.592223 11789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:02.603050 11794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:02.603050 11797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:02.603574 11789 server_base.cc:1034] running on GCE node
W20260929 17:58:02.603861 11795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:02.604569 11789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:02.605332 11789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:02.606546 11789 hybrid_clock.cc:648] HybridClock initialized: now 1790704682606528 us; error 41 us; skew 500 ppm
Sep 29 17:58:02 dist-test-slave-r7ph krb5kdc[11683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704682, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:02.610090 11789 init.cc:377] Logged in from keytab as kudu/127.7.118.1@KRBTEST.COM (short username kudu)
I20260929 17:58:02.611050 11789 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:02.611153 11789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:02.611409 11789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:02.613627 11789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-0/data/instance:
uuid: "2bffaf6c22674086b95eefed7676564e"
format_stamp: "Formatted at 2026-09-29 17:58:02 on dist-test-slave-r7ph"
server_key: "7bd302efae985c180082190beb5e4c0f"
server_key_iv: "b533d3609c852f7d4e99227312a32c9c"
server_key_version: "encryptionkey@0"
I20260929 17:58:02.614295 11789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance:
uuid: "2bffaf6c22674086b95eefed7676564e"
format_stamp: "Formatted at 2026-09-29 17:58:02 on dist-test-slave-r7ph"
server_key: "7bd302efae985c180082190beb5e4c0f"
server_key_iv: "b533d3609c852f7d4e99227312a32c9c"
server_key_version: "encryptionkey@0"
I20260929 17:58:02.618332 11789 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:58:02.621165 11803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:02.622658 11789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:58:02.622771 11789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "2bffaf6c22674086b95eefed7676564e"
format_stamp: "Formatted at 2026-09-29 17:58:02 on dist-test-slave-r7ph"
server_key: "7bd302efae985c180082190beb5e4c0f"
server_key_iv: "b533d3609c852f7d4e99227312a32c9c"
server_key_version: "encryptionkey@0"
I20260929 17:58:02.622879 11789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:02.638162 11789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:02.641841 11789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:02.642007 11789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:02.642675 11789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:02.643504 11789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:02.643551 11789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:02.643594 11789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:02.643610 11789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:02.653522 11789 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:43667
I20260929 17:58:02.653632 11916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:43667 every 8 connection(s)
I20260929 17:58:02.654202 11789 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 17:58:02.657660 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 11789
I20260929 17:58:02.657866 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance
I20260929 17:58:02.658226 7640 external_mini_cluster.cc:1428] Setting key 51f928c584b276322aa83321c1746625
WARNING: no policy specified for kudu/127.7.118.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.2@KRBTEST.COM" created.
Sep 29 17:58:02 dist-test-slave-r7ph krb5kdc[11683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704682, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:58:02.677766 11766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:02.656524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:53995 (local address 127.7.118.62:44813)
0929 17:58:02.656668 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:02.656677 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:02.657581 (+ 904us) server_negotiation.cc:407] Connection header received
0929 17:58:02.659289 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:02.659292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:02.659337 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:02.659418 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:02.661112 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:02.661710 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:02.663078 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:02.663328 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:02.663450 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:02.672811 (+ 9361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:02.672819 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:02.672826 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:02.672876 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:02.674850 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:02.675804 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:02.675807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:02.675809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:02.675913 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:02.676395 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:02.676397 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:02.676398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:02.676595 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:02.676672 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:02.677505 (+ 833us) server_negotiation.cc:299] Negotiation successful
0929 17:58:02.677634 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 17:58:02.679221 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:02.656848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:44813 (local address 127.7.118.1:53995)
0929 17:58:02.657371 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:02.657437 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:02.659059 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:02.659592 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:02.659599 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:02.660111 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:02.660928 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:02.660951 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:02.661854 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:02.661858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:02.662937 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:02.662948 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:02.663507 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:02.663511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:02.663706 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:02.670597 (+ 6891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:02.670631 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:02.672541 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:02.675055 (+ 2514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:02.675066 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:02.675091 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:02.675634 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:02.676031 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:02.676037 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:02.676042 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:02.676279 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:02.676707 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:02.676719 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:02.677154 (+ 435us) client_negotiation.cc:769] Sending connection context
0929 17:58:02.677406 (+ 252us) client_negotiation.cc:240] Negotiation successful
0929 17:58:02.677846 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":132,"threads_started":1}
I20260929 17:58:02.680734 11917 heartbeater.cc:347] Connected to a master server at 127.7.118.62:44813
I20260929 17:58:02.680984 11917 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:02.681528 11917 heartbeater.cc:511] Master 127.7.118.62:44813 requested a full tablet report, sending...
I20260929 17:58:02.683213 11727 ts_manager.cc:194] Registered new tserver with Master: 2bffaf6c22674086b95eefed7676564e (127.7.118.1:43667)
I20260929 17:58:02.684476 11727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.1@KRBTEST.COM'} at 127.7.118.1:53995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
I20260929 17:58:02.690567 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:0
--local_ip_for_outbound_sockets=127.7.118.2
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:44813
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:02.805218 11922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:02.805532 11922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:02.805581 11922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:02.810101 11922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:02.810184 11922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:02.810307 11922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 17:58:02.816040 11922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:44813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:02.817236 11922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:02.818442 11922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:02.828533 11928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:02.828969 11930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:02.829001 11927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:02.829197 11922 server_base.cc:1034] running on GCE node
I20260929 17:58:02.829631 11922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:02.830377 11922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:02.831607 11922 hybrid_clock.cc:648] HybridClock initialized: now 1790704682831566 us; error 57 us; skew 500 ppm
Sep 29 17:58:02 dist-test-slave-r7ph krb5kdc[11683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704682, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:02.834685 11922 init.cc:377] Logged in from keytab as kudu/127.7.118.2@KRBTEST.COM (short username kudu)
I20260929 17:58:02.835552 11922 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:02.835651 11922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:02.835882 11922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:02.838047 11922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-1/data/instance:
uuid: "1569f83066a34e0c82b07e1a22957f02"
format_stamp: "Formatted at 2026-09-29 17:58:02 on dist-test-slave-r7ph"
server_key: "0b78dd9a5a36e566bad49fc9be351b91"
server_key_iv: "83aa41ba7e48e72a7ae96bc81ca77c03"
server_key_version: "encryptionkey@0"
I20260929 17:58:02.838701 11922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance:
uuid: "1569f83066a34e0c82b07e1a22957f02"
format_stamp: "Formatted at 2026-09-29 17:58:02 on dist-test-slave-r7ph"
server_key: "0b78dd9a5a36e566bad49fc9be351b91"
server_key_iv: "83aa41ba7e48e72a7ae96bc81ca77c03"
server_key_version: "encryptionkey@0"
I20260929 17:58:02.842713 11922 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 17:58:02.845261 11936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:02.846793 11922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:58:02.846951 11922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "1569f83066a34e0c82b07e1a22957f02"
format_stamp: "Formatted at 2026-09-29 17:58:02 on dist-test-slave-r7ph"
server_key: "0b78dd9a5a36e566bad49fc9be351b91"
server_key_iv: "83aa41ba7e48e72a7ae96bc81ca77c03"
server_key_version: "encryptionkey@0"
I20260929 17:58:02.847082 11922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:02.868227 11922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:02.871876 11922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:02.872227 11922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:02.872963 11922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:02.873935 11922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:02.874006 11922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:02.874073 11922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:02.874125 11922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:02.884251 11922 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:45711
I20260929 17:58:02.884269 12049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:45711 every 8 connection(s)
I20260929 17:58:02.884927 11922 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 17:58:02.887413 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 11922
I20260929 17:58:02.887524 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance
I20260929 17:58:02.887847 7640 external_mini_cluster.cc:1428] Setting key 2152f7b0701ccf4c90feb5e3941f31bb
Sep 29 17:58:02 dist-test-slave-r7ph krb5kdc[11683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704682, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.118.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.3@KRBTEST.COM" created.
I20260929 17:58:02.900600 11766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:02.886893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:47101 (local address 127.7.118.62:44813)
0929 17:58:02.887147 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:02.887154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:02.888343 (+ 1189us) server_negotiation.cc:407] Connection header received
0929 17:58:02.889821 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:02.889825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:02.889880 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:02.889967 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:02.891265 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:02.891795 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:02.892999 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:02.893242 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:02.893363 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:02.896445 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:02.896455 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:02.896457 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:02.896485 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:02.897972 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:02.898660 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:02.898663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:02.898664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:02.898745 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:02.899142 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:02.899145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:02.899146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:02.899339 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:02.899461 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:02.900164 (+ 703us) server_negotiation.cc:299] Negotiation successful
0929 17:58:02.900325 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 17:58:02.901355 12052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:02.887478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:44813 (local address 127.7.118.2:47101)
0929 17:58:02.888141 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:02.888189 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:02.889603 (+ 1414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:02.890148 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:02.890155 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:02.890604 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:02.891113 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:02.891126 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:02.891933 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:02.891937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:02.892862 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:02.892872 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:02.893418 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:02.893422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:02.893606 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:02.894374 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:02.894392 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:02.896226 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:02.898129 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:02.898135 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:02.898151 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:02.898556 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:02.898858 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:02.898860 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:02.898862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:02.899045 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:02.899454 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:02.899459 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:02.899857 (+ 398us) client_negotiation.cc:769] Sending connection context
0929 17:58:02.900064 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 17:58:02.900373 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":163,"threads_started":1}
I20260929 17:58:02.902755 12050 heartbeater.cc:347] Connected to a master server at 127.7.118.62:44813
I20260929 17:58:02.903025 12050 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:02.903517 12050 heartbeater.cc:511] Master 127.7.118.62:44813 requested a full tablet report, sending...
I20260929 17:58:02.904757 11727 ts_manager.cc:194] Registered new tserver with Master: 1569f83066a34e0c82b07e1a22957f02 (127.7.118.2:45711)
I20260929 17:58:02.905411 11727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.2@KRBTEST.COM'} at 127.7.118.2:47101
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
I20260929 17:58:02.918181 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:0
--local_ip_for_outbound_sockets=127.7.118.3
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:44813
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:03.037541 12055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:03.037860 12055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:03.037916 12055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:03.042207 12055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:03.042333 12055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:03.042402 12055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 17:58:03.048305 12055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:44813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:03.049548 12055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:03.050808 12055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:03.062079 12063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:03.062129 12055 server_base.cc:1034] running on GCE node
W20260929 17:58:03.062069 12061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:03.062069 12060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:03.062888 12055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:03.063628 12055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:03.064869 12055 hybrid_clock.cc:648] HybridClock initialized: now 1790704683064840 us; error 43 us; skew 500 ppm
Sep 29 17:58:03 dist-test-slave-r7ph krb5kdc[11683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704683, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:03.068281 12055 init.cc:377] Logged in from keytab as kudu/127.7.118.3@KRBTEST.COM (short username kudu)
I20260929 17:58:03.069144 12055 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:03.069252 12055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:03.069502 12055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:03.071888 12055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-2/data/instance:
uuid: "e32e5e0ba2834b5b95dc3ba040aa292d"
format_stamp: "Formatted at 2026-09-29 17:58:03 on dist-test-slave-r7ph"
server_key: "5337b1a0a3b1b0903eb19881600354ae"
server_key_iv: "811d32ac5b424d8e8cd3ee00bb609b32"
server_key_version: "encryptionkey@0"
I20260929 17:58:03.072554 12055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance:
uuid: "e32e5e0ba2834b5b95dc3ba040aa292d"
format_stamp: "Formatted at 2026-09-29 17:58:03 on dist-test-slave-r7ph"
server_key: "5337b1a0a3b1b0903eb19881600354ae"
server_key_iv: "811d32ac5b424d8e8cd3ee00bb609b32"
server_key_version: "encryptionkey@0"
I20260929 17:58:03.077291 12055 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 17:58:03.080235 12069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:03.081874 12055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:58:03.082089 12055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "e32e5e0ba2834b5b95dc3ba040aa292d"
format_stamp: "Formatted at 2026-09-29 17:58:03 on dist-test-slave-r7ph"
server_key: "5337b1a0a3b1b0903eb19881600354ae"
server_key_iv: "811d32ac5b424d8e8cd3ee00bb609b32"
server_key_version: "encryptionkey@0"
I20260929 17:58:03.082247 12055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:03.098856 12055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:03.102756 12055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:03.103006 12055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:03.103688 12055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:03.104723 12055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:03.104797 12055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:03.104879 12055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:03.104921 12055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:03.114959 12055 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:41021
I20260929 17:58:03.114976 12182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:41021 every 8 connection(s)
I20260929 17:58:03.115700 12055 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 17:58:03.125602 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 12055
I20260929 17:58:03.125725 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance
Sep 29 17:58:03 dist-test-slave-r7ph krb5kdc[11683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704683, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:58:03.126179 7640 external_mini_cluster.cc:1428] Setting key 791d9b8a899b9aba149bb2ab4a297e84
I20260929 17:58:03.131150 11766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:03.117649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:46935 (local address 127.7.118.62:44813)
0929 17:58:03.117812 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:03.117816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:03.118850 (+ 1034us) server_negotiation.cc:407] Connection header received
0929 17:58:03.120288 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:03.120292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:03.120352 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:03.120436 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:03.121764 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.122421 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.123656 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.123884 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.124011 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:03.126999 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:03.127003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:03.127005 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:03.127034 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:03.128803 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:03.129370 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:03.129374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:03.129376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:03.129476 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:03.129853 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:03.129857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:03.129859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:03.130070 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:03.130184 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:03.130817 (+ 633us) server_negotiation.cc:299] Negotiation successful
0929 17:58:03.130968 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 17:58:03.132169 12185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:03.118041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:44813 (local address 127.7.118.3:46935)
0929 17:58:03.118659 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:03.118709 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:03.120017 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:03.120631 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:03.120639 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:03.121033 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:03.121591 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:03.121607 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.122569 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.122572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:03.123513 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:03.123524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.124063 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.124067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:03.124262 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:03.124947 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:03.124966 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:03.126715 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:03.128930 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:03.128936 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:03.128956 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:03.129259 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:03.129573 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:03.129577 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:03.129579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:03.129751 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:03.130199 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:03.130211 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:03.130550 (+ 339us) client_negotiation.cc:769] Sending connection context
0929 17:58:03.130760 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 17:58:03.131145 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":193,"threads_started":1}
I20260929 17:58:03.133606 12183 heartbeater.cc:347] Connected to a master server at 127.7.118.62:44813
I20260929 17:58:03.133869 12183 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:03.134486 12183 heartbeater.cc:511] Master 127.7.118.62:44813 requested a full tablet report, sending...
I20260929 17:58:03.135721 11727 ts_manager.cc:194] Registered new tserver with Master: e32e5e0ba2834b5b95dc3ba040aa292d (127.7.118.3:41021)
I20260929 17:58:03.136379 11727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.3@KRBTEST.COM'} at 127.7.118.3:46935
I20260929 17:58:03.140942 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:58:03.152110 11766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:03.143046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52262 (local address 127.7.118.62:44813)
0929 17:58:03.143223 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:03.143228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:03.143398 (+ 170us) server_negotiation.cc:407] Connection header received
0929 17:58:03.143604 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:03.143607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:03.143664 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:03.143753 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:03.144517 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.145193 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.146104 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.146452 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.146575 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:03.148074 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:03.148079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:03.148082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:03.148119 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:03.149874 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:03.150506 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:03.150510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:03.150512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:03.150674 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:03.151022 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:03.151026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:03.151028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:03.151303 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:03.151477 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:03.151777 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 17:58:03.151925 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 17:58:03.156415 11727 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:58:03.159667 11727 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:58:03.176651 12194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:03.169244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:45711 (local address 127.0.0.1:45386)
0929 17:58:03.170333 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:03.170397 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:03.170553 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:03.171349 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:03.171356 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:03.171379 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:03.171548 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:03.171559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.173317 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.173320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:03.174905 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:03.174927 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.175956 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.175959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:03.176097 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:03.176167 (+ 70us) client_negotiation.cc:769] Sending connection context
0929 17:58:03.176285 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 17:58:03.176460 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"mutex_wait_us":105,"thread_start_us":134,"threads_started":1}
I20260929 17:58:03.176713 12199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:03.170423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51892 (local address 127.7.118.3:41021)
0929 17:58:03.170888 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:03.170901 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:03.171054 (+ 153us) server_negotiation.cc:407] Connection header received
0929 17:58:03.171226 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:03.171233 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:03.171445 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:03.171614 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:03.171858 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.173131 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.174940 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.175760 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.175905 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:03.176101 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:03.176428 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 17:58:03.176552 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":178,"threads_started":1}
I20260929 17:58:03.177533 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:03.169473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45386 (local address 127.7.118.2:45711)
0929 17:58:03.170195 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:03.170210 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:03.170706 (+ 496us) server_negotiation.cc:407] Connection header received
0929 17:58:03.170814 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:03.170820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:03.171111 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:03.171326 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:03.171693 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.173132 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.175051 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.175794 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.175926 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:03.176095 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:03.177253 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0929 17:58:03.177396 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":111,"threads_started":1}
I20260929 17:58:03.177953 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:03.169535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:41021 (local address 127.0.0.1:51892)
0929 17:58:03.170924 (+ 1389us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:03.170941 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:03.171056 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:03.171608 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:03.171611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:03.171629 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:03.171721 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:03.171727 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.173317 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.173320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:03.174816 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:03.174828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.175907 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.175911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:03.176087 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:03.176155 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 17:58:03.176319 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 17:58:03.177816 (+ 1497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1314,"mutex_wait_us":1509,"thread_start_us":117,"threads_started":1}
I20260929 17:58:03.178071 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:03.169669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:43667 (local address 127.0.0.1:35782)
0929 17:58:03.171034 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:03.171047 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:03.171141 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:03.171804 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:03.171808 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:03.171827 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:03.171915 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:03.171919 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.172968 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.172973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:03.176152 (+ 3179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:03.176161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.177568 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.177571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:03.177680 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:03.177706 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 17:58:03.177798 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 17:58:03.177882 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1306,"mutex_wait_us":1467,"thread_start_us":106,"threads_started":1}
I20260929 17:58:03.178942 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:03.170396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35782 (local address 127.7.118.1:43667)
0929 17:58:03.170920 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:03.170926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:03.171114 (+ 188us) server_negotiation.cc:407] Connection header received
0929 17:58:03.171271 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:03.171277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:03.171476 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:03.171653 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:03.172026 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.172807 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.176308 (+ 3501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.177274 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.177397 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:03.177514 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:03.177861 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 17:58:03.177976 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":99,"threads_started":1}
I20260929 17:58:03.181002 11984 tablet_service.cc:1467] Processing CreateTablet for tablet 86127411606046459bb3cff5a8f741b5 (DEFAULT_TABLE table=test-table [id=38d773238a9944df95ebd5664c258f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:58:03.181002 12117 tablet_service.cc:1467] Processing CreateTablet for tablet 86127411606046459bb3cff5a8f741b5 (DEFAULT_TABLE table=test-table [id=38d773238a9944df95ebd5664c258f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:58:03.182322 11984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86127411606046459bb3cff5a8f741b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:03.182322 12117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86127411606046459bb3cff5a8f741b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:03.184885 11851 tablet_service.cc:1467] Processing CreateTablet for tablet 86127411606046459bb3cff5a8f741b5 (DEFAULT_TABLE table=test-table [id=38d773238a9944df95ebd5664c258f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:58:03.186015 11851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86127411606046459bb3cff5a8f741b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:03.190290 12200 tablet_bootstrap.cc:492] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02: Bootstrap starting.
I20260929 17:58:03.190773 12201 tablet_bootstrap.cc:492] T 86127411606046459bb3cff5a8f741b5 P e32e5e0ba2834b5b95dc3ba040aa292d: Bootstrap starting.
I20260929 17:58:03.192888 12202 tablet_bootstrap.cc:492] T 86127411606046459bb3cff5a8f741b5 P 2bffaf6c22674086b95eefed7676564e: Bootstrap starting.
I20260929 17:58:03.193362 12201 tablet_bootstrap.cc:654] T 86127411606046459bb3cff5a8f741b5 P e32e5e0ba2834b5b95dc3ba040aa292d: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:03.194552 12201 log.cc:824] T 86127411606046459bb3cff5a8f741b5 P e32e5e0ba2834b5b95dc3ba040aa292d: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:03.195199 12202 tablet_bootstrap.cc:654] T 86127411606046459bb3cff5a8f741b5 P 2bffaf6c22674086b95eefed7676564e: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:03.195982 12202 log.cc:824] T 86127411606046459bb3cff5a8f741b5 P 2bffaf6c22674086b95eefed7676564e: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:03.197077 12200 tablet_bootstrap.cc:654] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:03.198091 12200 log.cc:824] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:03.198199 12202 tablet_bootstrap.cc:492] T 86127411606046459bb3cff5a8f741b5 P 2bffaf6c22674086b95eefed7676564e: No bootstrap required, opened a new log
I20260929 17:58:03.198459 12202 ts_tablet_manager.cc:1397] T 86127411606046459bb3cff5a8f741b5 P 2bffaf6c22674086b95eefed7676564e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260929 17:58:03.200022 12201 tablet_bootstrap.cc:492] T 86127411606046459bb3cff5a8f741b5 P e32e5e0ba2834b5b95dc3ba040aa292d: No bootstrap required, opened a new log
I20260929 17:58:03.200037 12200 tablet_bootstrap.cc:492] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02: No bootstrap required, opened a new log
I20260929 17:58:03.200246 12201 ts_tablet_manager.cc:1397] T 86127411606046459bb3cff5a8f741b5 P e32e5e0ba2834b5b95dc3ba040aa292d: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260929 17:58:03.200246 12200 ts_tablet_manager.cc:1397] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260929 17:58:03.202399 12202 raft_consensus.cc:348] T 86127411606046459bb3cff5a8f741b5 P 2bffaf6c22674086b95eefed7676564e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1569f83066a34e0c82b07e1a22957f02" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 45711 } } peers { permanent_uuid: "e32e5e0ba2834b5b95dc3ba040aa292d" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 41021 } } peers { permanent_uuid: "2bffaf6c22674086b95eefed7676564e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 43667 } }
I20260929 17:58:03.202670 12202 raft_consensus.cc:374] T 86127411606046459bb3cff5a8f741b5 P 2bffaf6c22674086b95eefed7676564e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:03.202742 12202 raft_consensus.cc:729] T 86127411606046459bb3cff5a8f741b5 P 2bffaf6c22674086b95eefed7676564e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bffaf6c22674086b95eefed7676564e, State: Initialized, Role: FOLLOWER
I20260929 17:58:03.203228 12202 consensus_queue.cc:260] T 86127411606046459bb3cff5a8f741b5 P 2bffaf6c22674086b95eefed7676564e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1569f83066a34e0c82b07e1a22957f02" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 45711 } } peers { permanent_uuid: "e32e5e0ba2834b5b95dc3ba040aa292d" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 41021 } } peers { permanent_uuid: "2bffaf6c22674086b95eefed7676564e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 43667 } }
I20260929 17:58:03.203929 11917 heartbeater.cc:503] Master 127.7.118.62:44813 was elected leader, sending a full tablet report...
I20260929 17:58:03.204459 12202 ts_tablet_manager.cc:1428] T 86127411606046459bb3cff5a8f741b5 P 2bffaf6c22674086b95eefed7676564e: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260929 17:58:03.204288 12201 raft_consensus.cc:348] T 86127411606046459bb3cff5a8f741b5 P e32e5e0ba2834b5b95dc3ba040aa292d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1569f83066a34e0c82b07e1a22957f02" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 45711 } } peers { permanent_uuid: "e32e5e0ba2834b5b95dc3ba040aa292d" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 41021 } } peers { permanent_uuid: "2bffaf6c22674086b95eefed7676564e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 43667 } }
I20260929 17:58:03.204288 12200 raft_consensus.cc:348] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1569f83066a34e0c82b07e1a22957f02" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 45711 } } peers { permanent_uuid: "e32e5e0ba2834b5b95dc3ba040aa292d" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 41021 } } peers { permanent_uuid: "2bffaf6c22674086b95eefed7676564e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 43667 } }
I20260929 17:58:03.204576 12200 raft_consensus.cc:374] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:03.204576 12201 raft_consensus.cc:374] T 86127411606046459bb3cff5a8f741b5 P e32e5e0ba2834b5b95dc3ba040aa292d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:03.204628 12201 raft_consensus.cc:729] T 86127411606046459bb3cff5a8f741b5 P e32e5e0ba2834b5b95dc3ba040aa292d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e32e5e0ba2834b5b95dc3ba040aa292d, State: Initialized, Role: FOLLOWER
I20260929 17:58:03.204628 12200 raft_consensus.cc:729] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1569f83066a34e0c82b07e1a22957f02, State: Initialized, Role: FOLLOWER
I20260929 17:58:03.205107 12200 consensus_queue.cc:260] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1569f83066a34e0c82b07e1a22957f02" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 45711 } } peers { permanent_uuid: "e32e5e0ba2834b5b95dc3ba040aa292d" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 41021 } } peers { permanent_uuid: "2bffaf6c22674086b95eefed7676564e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 43667 } }
I20260929 17:58:03.205107 12201 consensus_queue.cc:260] T 86127411606046459bb3cff5a8f741b5 P e32e5e0ba2834b5b95dc3ba040aa292d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1569f83066a34e0c82b07e1a22957f02" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 45711 } } peers { permanent_uuid: "e32e5e0ba2834b5b95dc3ba040aa292d" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 41021 } } peers { permanent_uuid: "2bffaf6c22674086b95eefed7676564e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 43667 } }
I20260929 17:58:03.206176 12201 ts_tablet_manager.cc:1428] T 86127411606046459bb3cff5a8f741b5 P e32e5e0ba2834b5b95dc3ba040aa292d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 17:58:03.206176 12200 ts_tablet_manager.cc:1428] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 17:58:03.206475 12183 heartbeater.cc:503] Master 127.7.118.62:44813 was elected leader, sending a full tablet report...
I20260929 17:58:03.206573 12050 heartbeater.cc:503] Master 127.7.118.62:44813 was elected leader, sending a full tablet report...
I20260929 17:58:03.262845 12206 raft_consensus.cc:482] T 86127411606046459bb3cff5a8f741b5 P 2bffaf6c22674086b95eefed7676564e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:58:03.263101 12206 raft_consensus.cc:504] T 86127411606046459bb3cff5a8f741b5 P 2bffaf6c22674086b95eefed7676564e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1569f83066a34e0c82b07e1a22957f02" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 45711 } } peers { permanent_uuid: "e32e5e0ba2834b5b95dc3ba040aa292d" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 41021 } } peers { permanent_uuid: "2bffaf6c22674086b95eefed7676564e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 43667 } }
I20260929 17:58:03.263592 12208 raft_consensus.cc:482] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:58:03.263785 12208 raft_consensus.cc:504] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1569f83066a34e0c82b07e1a22957f02" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 45711 } } peers { permanent_uuid: "e32e5e0ba2834b5b95dc3ba040aa292d" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 41021 } } peers { permanent_uuid: "2bffaf6c22674086b95eefed7676564e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 43667 } }
I20260929 17:58:03.264685 12206 leader_election.cc:288] T 86127411606046459bb3cff5a8f741b5 P 2bffaf6c22674086b95eefed7676564e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1569f83066a34e0c82b07e1a22957f02 (127.7.118.2:45711), e32e5e0ba2834b5b95dc3ba040aa292d (127.7.118.3:41021)
I20260929 17:58:03.266113 12208 leader_election.cc:288] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e32e5e0ba2834b5b95dc3ba040aa292d (127.7.118.3:41021), 2bffaf6c22674086b95eefed7676564e (127.7.118.1:43667)
I20260929 17:58:03.270756 12052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:03.265089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:41021 (local address 127.7.118.2:60645)
0929 17:58:03.265419 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:03.265436 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:03.265553 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:03.266005 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:03.266009 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:03.266031 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:03.266154 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:03.266161 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.267360 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.267364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:03.268834 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:03.268845 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.270098 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.270103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:03.270319 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:03.270406 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 17:58:03.270492 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 17:58:03.270584 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":69}
I20260929 17:58:03.270743 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:03.266057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:56095 (local address 127.7.118.1:43667)
0929 17:58:03.266338 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:03.266343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:03.266577 (+ 234us) server_negotiation.cc:407] Connection header received
0929 17:58:03.266744 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:03.266749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:03.266793 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:03.266838 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:03.267195 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.267826 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.269311 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.269936 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.270028 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:03.270098 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:03.270542 (+ 444us) server_negotiation.cc:299] Negotiation successful
0929 17:58:03.270619 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213}
I20260929 17:58:03.271150 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:03.265008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:41021 (local address 127.7.118.1:40963)
0929 17:58:03.265623 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:03.265640 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:03.265739 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:03.266079 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:03.266082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:03.266099 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:03.266229 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:03.266238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.267360 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.267364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:03.268801 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:03.268815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.270705 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.270709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:03.270811 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:03.270839 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 17:58:03.270912 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 17:58:03.270984 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":129,"threads_started":1}
I20260929 17:58:03.271414 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:03.265008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:45711 (local address 127.7.118.1:56683)
0929 17:58:03.265771 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:03.265787 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:03.265946 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:03.266422 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:03.266424 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:03.266455 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:03.266596 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:03.266603 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.268713 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.268716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:03.269711 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:03.269719 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.270764 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.270767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:03.270939 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:03.270966 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 17:58:03.271214 (+ 248us) client_negotiation.cc:240] Negotiation successful
0929 17:58:03.271291 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":18,"thread_start_us":109,"threads_started":1}
I20260929 17:58:03.271476 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:03.265376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:56683 (local address 127.7.118.2:45711)
0929 17:58:03.265481 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:03.265485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:03.265864 (+ 379us) server_negotiation.cc:407] Connection header received
0929 17:58:03.266073 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:03.266078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:03.266124 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:03.266176 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:03.266719 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.267487 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.269839 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.270565 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.270650 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:03.270704 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:03.271083 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 17:58:03.271153 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":39}
I20260929 17:58:03.271194 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:03.265703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:43667 (local address 127.7.118.2:56095)
0929 17:58:03.266496 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:03.266510 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:03.266634 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:03.266960 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:03.266963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:03.266978 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:03.267083 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:03.267088 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.267974 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.267977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:03.269182 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:03.269192 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.270148 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.270151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:03.270355 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:03.270409 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 17:58:03.270928 (+ 519us) client_negotiation.cc:240] Negotiation successful
0929 17:58:03.271024 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"mutex_wait_us":109,"thread_start_us":108,"threads_started":1}
I20260929 17:58:03.271572 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:03.265208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:40963 (local address 127.7.118.3:41021)
0929 17:58:03.265578 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:03.265589 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:03.265828 (+ 239us) server_negotiation.cc:407] Connection header received
0929 17:58:03.265870 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:03.265873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:03.265905 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:03.265976 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:03.266397 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.267221 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.268973 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.270463 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.270565 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:03.270643 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:03.271312 (+ 669us) server_negotiation.cc:299] Negotiation successful
0929 17:58:03.271389 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":657,"server-negotiator.queue_time_us":307,"thread_start_us":116,"threads_started":1}
I20260929 17:58:03.272147 12199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:03.265104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:60645 (local address 127.7.118.3:41021)
0929 17:58:03.265411 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:03.265417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:03.265654 (+ 237us) server_negotiation.cc:407] Connection header received
0929 17:58:03.265720 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:03.265723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:03.265798 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:03.265871 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:03.266320 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.267214 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.268973 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.269822 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.269937 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:03.270033 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:03.271960 (+ 1927us) server_negotiation.cc:299] Negotiation successful
0929 17:58:03.272028 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":113,"server-negotiator.queue_time_us":208}
I20260929 17:58:03.272249 12004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86127411606046459bb3cff5a8f741b5" candidate_uuid: "2bffaf6c22674086b95eefed7676564e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1569f83066a34e0c82b07e1a22957f02" is_pre_election: true
I20260929 17:58:03.272228 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86127411606046459bb3cff5a8f741b5" candidate_uuid: "2bffaf6c22674086b95eefed7676564e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e32e5e0ba2834b5b95dc3ba040aa292d" is_pre_election: true
I20260929 17:58:03.272285 11871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86127411606046459bb3cff5a8f741b5" candidate_uuid: "1569f83066a34e0c82b07e1a22957f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bffaf6c22674086b95eefed7676564e" is_pre_election: true
I20260929 17:58:03.272495 11871 raft_consensus.cc:2454] T 86127411606046459bb3cff5a8f741b5 P 2bffaf6c22674086b95eefed7676564e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1569f83066a34e0c82b07e1a22957f02 in term 0.
I20260929 17:58:03.272492 12004 raft_consensus.cc:2454] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bffaf6c22674086b95eefed7676564e in term 0.
I20260929 17:58:03.272545 12137 raft_consensus.cc:2454] T 86127411606046459bb3cff5a8f741b5 P e32e5e0ba2834b5b95dc3ba040aa292d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bffaf6c22674086b95eefed7676564e in term 0.
I20260929 17:58:03.272610 12136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86127411606046459bb3cff5a8f741b5" candidate_uuid: "1569f83066a34e0c82b07e1a22957f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e32e5e0ba2834b5b95dc3ba040aa292d" is_pre_election: true
I20260929 17:58:03.272711 12136 raft_consensus.cc:2454] T 86127411606046459bb3cff5a8f741b5 P e32e5e0ba2834b5b95dc3ba040aa292d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1569f83066a34e0c82b07e1a22957f02 in term 0.
I20260929 17:58:03.273005 11937 leader_election.cc:302] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1569f83066a34e0c82b07e1a22957f02, 2bffaf6c22674086b95eefed7676564e; no voters:
I20260929 17:58:03.273003 11806 leader_election.cc:302] T 86127411606046459bb3cff5a8f741b5 P 2bffaf6c22674086b95eefed7676564e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1569f83066a34e0c82b07e1a22957f02, 2bffaf6c22674086b95eefed7676564e, e32e5e0ba2834b5b95dc3ba040aa292d; no voters:
I20260929 17:58:03.273324 12208 raft_consensus.cc:2781] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:58:03.273409 12208 raft_consensus.cc:482] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:58:03.273483 12208 raft_consensus.cc:3037] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:03.273538 12206 raft_consensus.cc:2781] T 86127411606046459bb3cff5a8f741b5 P 2bffaf6c22674086b95eefed7676564e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:58:03.273636 12206 raft_consensus.cc:482] T 86127411606046459bb3cff5a8f741b5 P 2bffaf6c22674086b95eefed7676564e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:58:03.273699 12206 raft_consensus.cc:3037] T 86127411606046459bb3cff5a8f741b5 P 2bffaf6c22674086b95eefed7676564e [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:03.274787 12208 raft_consensus.cc:504] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1569f83066a34e0c82b07e1a22957f02" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 45711 } } peers { permanent_uuid: "e32e5e0ba2834b5b95dc3ba040aa292d" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 41021 } } peers { permanent_uuid: "2bffaf6c22674086b95eefed7676564e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 43667 } }
I20260929 17:58:03.274983 12206 raft_consensus.cc:504] T 86127411606046459bb3cff5a8f741b5 P 2bffaf6c22674086b95eefed7676564e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1569f83066a34e0c82b07e1a22957f02" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 45711 } } peers { permanent_uuid: "e32e5e0ba2834b5b95dc3ba040aa292d" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 41021 } } peers { permanent_uuid: "2bffaf6c22674086b95eefed7676564e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 43667 } }
I20260929 17:58:03.275224 12208 leader_election.cc:288] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02 [CANDIDATE]: Term 1 election: Requested vote from peers e32e5e0ba2834b5b95dc3ba040aa292d (127.7.118.3:41021), 2bffaf6c22674086b95eefed7676564e (127.7.118.1:43667)
I20260929 17:58:03.275439 12206 leader_election.cc:288] T 86127411606046459bb3cff5a8f741b5 P 2bffaf6c22674086b95eefed7676564e [CANDIDATE]: Term 1 election: Requested vote from peers 1569f83066a34e0c82b07e1a22957f02 (127.7.118.2:45711), e32e5e0ba2834b5b95dc3ba040aa292d (127.7.118.3:41021)
I20260929 17:58:03.275648 11871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86127411606046459bb3cff5a8f741b5" candidate_uuid: "1569f83066a34e0c82b07e1a22957f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bffaf6c22674086b95eefed7676564e"
I20260929 17:58:03.275655 12136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86127411606046459bb3cff5a8f741b5" candidate_uuid: "1569f83066a34e0c82b07e1a22957f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e32e5e0ba2834b5b95dc3ba040aa292d"
I20260929 17:58:03.275800 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86127411606046459bb3cff5a8f741b5" candidate_uuid: "2bffaf6c22674086b95eefed7676564e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e32e5e0ba2834b5b95dc3ba040aa292d"
I20260929 17:58:03.275846 11871 raft_consensus.cc:2379] T 86127411606046459bb3cff5a8f741b5 P 2bffaf6c22674086b95eefed7676564e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1569f83066a34e0c82b07e1a22957f02 in current term 1: Already voted for candidate 2bffaf6c22674086b95eefed7676564e in this term.
I20260929 17:58:03.275863 12136 raft_consensus.cc:3037] T 86127411606046459bb3cff5a8f741b5 P e32e5e0ba2834b5b95dc3ba040aa292d [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:03.276677 12004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86127411606046459bb3cff5a8f741b5" candidate_uuid: "2bffaf6c22674086b95eefed7676564e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1569f83066a34e0c82b07e1a22957f02"
I20260929 17:58:03.276881 12004 raft_consensus.cc:2379] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2bffaf6c22674086b95eefed7676564e in current term 1: Already voted for candidate 1569f83066a34e0c82b07e1a22957f02 in this term.
I20260929 17:58:03.277433 12136 raft_consensus.cc:2454] T 86127411606046459bb3cff5a8f741b5 P e32e5e0ba2834b5b95dc3ba040aa292d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1569f83066a34e0c82b07e1a22957f02 in term 1.
I20260929 17:58:03.277482 11805 leader_election.cc:302] T 86127411606046459bb3cff5a8f741b5 P 2bffaf6c22674086b95eefed7676564e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2bffaf6c22674086b95eefed7676564e; no voters: 1569f83066a34e0c82b07e1a22957f02, e32e5e0ba2834b5b95dc3ba040aa292d
I20260929 17:58:03.277683 12206 raft_consensus.cc:2726] T 86127411606046459bb3cff5a8f741b5 P 2bffaf6c22674086b95eefed7676564e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 17:58:03.277830 11939 leader_election.cc:302] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1569f83066a34e0c82b07e1a22957f02, e32e5e0ba2834b5b95dc3ba040aa292d; no voters: 2bffaf6c22674086b95eefed7676564e
I20260929 17:58:03.278025 12208 raft_consensus.cc:2781] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:58:03.278302 12208 raft_consensus.cc:686] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02 [term 1 LEADER]: Becoming Leader. State: Replica: 1569f83066a34e0c82b07e1a22957f02, State: Running, Role: LEADER
I20260929 17:58:03.278666 12208 consensus_queue.cc:237] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1569f83066a34e0c82b07e1a22957f02" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 45711 } } peers { permanent_uuid: "e32e5e0ba2834b5b95dc3ba040aa292d" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 41021 } } peers { permanent_uuid: "2bffaf6c22674086b95eefed7676564e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 43667 } }
I20260929 17:58:03.282240 11725 catalog_manager.cc:5495] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1569f83066a34e0c82b07e1a22957f02 (127.7.118.2). New cstate: current_term: 1 leader_uuid: "1569f83066a34e0c82b07e1a22957f02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1569f83066a34e0c82b07e1a22957f02" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 45711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e32e5e0ba2834b5b95dc3ba040aa292d" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 41021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bffaf6c22674086b95eefed7676564e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 43667 } health_report { overall_health: UNKNOWN } } }
I20260929 17:58:03.302543 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:03.298564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45388 (local address 127.7.118.2:45711)
0929 17:58:03.298691 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:03.298694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:03.298709 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:58:03.298809 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:03.298811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:03.298869 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:03.298972 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:58:03.299274 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.299785 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.300708 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:03.300938 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:03.301023 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:03.301462 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:58:03.301902 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:58:03.302007 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:03.302286 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 17:58:03.302400 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 17:58:03.309391 12136 raft_consensus.cc:1261] T 86127411606046459bb3cff5a8f741b5 P e32e5e0ba2834b5b95dc3ba040aa292d [term 1 FOLLOWER]: Refusing update from remote peer 1569f83066a34e0c82b07e1a22957f02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:58:03.309393 11871 raft_consensus.cc:1261] T 86127411606046459bb3cff5a8f741b5 P 2bffaf6c22674086b95eefed7676564e [term 1 FOLLOWER]: Refusing update from remote peer 1569f83066a34e0c82b07e1a22957f02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:58:03.310125 12213 consensus_queue.cc:1035] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bffaf6c22674086b95eefed7676564e" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:58:03.310371 12208 consensus_queue.cc:1035] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e32e5e0ba2834b5b95dc3ba040aa292d" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:58:03.331235 11726 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:58:03.331418 11726 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:58:03.334295 11726 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee83326e0eee457eb2cdd39e325ec16c: Sending DeleteTablet for 3 replicas of tablet 86127411606046459bb3cff5a8f741b5
I20260929 17:58:03.335347 11984 tablet_service.cc:1514] Processing DeleteTablet for tablet 86127411606046459bb3cff5a8f741b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:58:03 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:45386
I20260929 17:58:03.335391 11851 tablet_service.cc:1514] Processing DeleteTablet for tablet 86127411606046459bb3cff5a8f741b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:58:03 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:35782
I20260929 17:58:03.335994 12117 tablet_service.cc:1514] Processing DeleteTablet for tablet 86127411606046459bb3cff5a8f741b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:58:03 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:51892
I20260929 17:58:03.336220 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 11789
I20260929 17:58:03.337893 12228 tablet_replica.cc:330] T 86127411606046459bb3cff5a8f741b5 P e32e5e0ba2834b5b95dc3ba040aa292d: stopping tablet replica
I20260929 17:58:03.338366 12228 raft_consensus.cc:2229] T 86127411606046459bb3cff5a8f741b5 P e32e5e0ba2834b5b95dc3ba040aa292d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:58:03.338472 12226 tablet_replica.cc:330] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02: stopping tablet replica
I20260929 17:58:03.338779 12226 raft_consensus.cc:2229] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:58:03.338892 12228 raft_consensus.cc:2258] T 86127411606046459bb3cff5a8f741b5 P e32e5e0ba2834b5b95dc3ba040aa292d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:58:03.339242 12226 raft_consensus.cc:2258] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:58:03.340760 12228 ts_tablet_manager.cc:1905] T 86127411606046459bb3cff5a8f741b5 P e32e5e0ba2834b5b95dc3ba040aa292d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:58:03.340760 12226 ts_tablet_manager.cc:1905] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:58:03.344059 12226 ts_tablet_manager.cc:1918] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:58:03.344053 12228 ts_tablet_manager.cc:1918] T 86127411606046459bb3cff5a8f741b5 P e32e5e0ba2834b5b95dc3ba040aa292d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:58:03.344180 12228 log.cc:1196] T 86127411606046459bb3cff5a8f741b5 P e32e5e0ba2834b5b95dc3ba040aa292d: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-2/wal/wals/86127411606046459bb3cff5a8f741b5
I20260929 17:58:03.344352 12226 log.cc:1196] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704657566865-7640-0/minicluster-data/ts-1/wal/wals/86127411606046459bb3cff5a8f741b5
I20260929 17:58:03.344568 12228 ts_tablet_manager.cc:1939] T 86127411606046459bb3cff5a8f741b5 P e32e5e0ba2834b5b95dc3ba040aa292d: Deleting consensus metadata
I20260929 17:58:03.344754 12226 ts_tablet_manager.cc:1939] T 86127411606046459bb3cff5a8f741b5 P 1569f83066a34e0c82b07e1a22957f02: Deleting consensus metadata
I20260929 17:58:03.346014 11712 catalog_manager.cc:4841] TS 1569f83066a34e0c82b07e1a22957f02 (127.7.118.2:45711): tablet 86127411606046459bb3cff5a8f741b5 (table test-table [id=38d773238a9944df95ebd5664c258f0c]) successfully deleted
W20260929 17:58:03.346390 11711 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.118.1:43667 (error 108)
I20260929 17:58:03.346017 11713 catalog_manager.cc:4841] TS e32e5e0ba2834b5b95dc3ba040aa292d (127.7.118.3:41021): tablet 86127411606046459bb3cff5a8f741b5 (table test-table [id=38d773238a9944df95ebd5664c258f0c]) successfully deleted
I20260929 17:58:03.346514 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 11922
I20260929 17:58:03.348052 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:03.347771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:43667 (local address 127.0.0.1:35792)
0929 17:58:03.347903 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:03.347973 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.1:43667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 17:58:03.348901 11711 catalog_manager.cc:4584] TS 2bffaf6c22674086b95eefed7676564e (127.7.118.1:43667): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 86127411606046459bb3cff5a8f741b5: Network error: Client connection negotiation failed: client connection to 127.7.118.1:43667: connect: Connection refused (error 111)
I20260929 17:58:03.353317 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 12055
I20260929 17:58:03.360515 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 11696
2026-09-29T17:58:03Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2432 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T17:58:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:58:03Z Disabled control of system clock
I20260929 17:58:03.394035 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:41979
--webserver_interface=127.7.118.62
--webserver_port=0
--builtin_ntp_servers=127.7.118.20:46463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:41979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 17:58:03.520310 12233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:03.520625 12233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:03.520665 12233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:03.525054 12233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:58:03.525142 12233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:03.525158 12233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:03.525177 12233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:58:03.525192 12233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:58:03.530737 12233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:46463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:41979
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:41979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:03.532032 12233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:03.533602 12233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:03.543169 12238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:03.543169 12239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:03.543164 12241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:03.543215 12233 server_base.cc:1034] running on GCE node
I20260929 17:58:03.544163 12233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:03.545408 12233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:03.546619 12233 hybrid_clock.cc:648] HybridClock initialized: now 1790704683546582 us; error 54 us; skew 500 ppm
I20260929 17:58:03.548811 12233 webserver.cc:460] Webserver started at http://127.7.118.62:40531/ using document root <none> and password file <none>
I20260929 17:58:03.549430 12233 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:03.549500 12233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:03.549679 12233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:03.552055 12233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "1128ed8b21d44107bfc2b87ca453b572"
format_stamp: "Formatted at 2026-09-29 17:58:03 on dist-test-slave-r7ph"
server_key: "8113a278f4e1ac50b61dbb3d4109846f"
server_key_iv: "d48be2eb91d0d010061d0bcb8d184b0f"
server_key_version: "encryptionkey@0"
I20260929 17:58:03.552659 12233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "1128ed8b21d44107bfc2b87ca453b572"
format_stamp: "Formatted at 2026-09-29 17:58:03 on dist-test-slave-r7ph"
server_key: "8113a278f4e1ac50b61dbb3d4109846f"
server_key_iv: "d48be2eb91d0d010061d0bcb8d184b0f"
server_key_version: "encryptionkey@0"
I20260929 17:58:03.556820 12233 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260929 17:58:03.559675 12247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:03.561188 12233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 17:58:03.561316 12233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "1128ed8b21d44107bfc2b87ca453b572"
format_stamp: "Formatted at 2026-09-29 17:58:03 on dist-test-slave-r7ph"
server_key: "8113a278f4e1ac50b61dbb3d4109846f"
server_key_iv: "d48be2eb91d0d010061d0bcb8d184b0f"
server_key_version: "encryptionkey@0"
I20260929 17:58:03.561434 12233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:03.566951 12233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:03.567612 12233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:03.567767 12233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:03.575484 12233 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:41979
I20260929 17:58:03.575496 12299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:41979 every 8 connection(s)
I20260929 17:58:03.576560 12233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 17:58:03.580118 12300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:03.581219 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 12233
I20260929 17:58:03.581377 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 17:58:03.581753 7640 external_mini_cluster.cc:1428] Setting key ab398852decb867a9c3791176b23ae45
I20260929 17:58:03.587548 12300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1128ed8b21d44107bfc2b87ca453b572: Bootstrap starting.
I20260929 17:58:03.588752 12303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:03.583433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49764 (local address 127.7.118.62:41979)
0929 17:58:03.584232 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:03.584255 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:03.584318 (+ 63us) server_negotiation.cc:407] Connection header received
0929 17:58:03.585238 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:03.585264 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:03.585784 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:03.586217 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:03.587084 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:03.587090 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:03.587107 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:58:03.587120 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:03.587191 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:58:03.587214 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:03.587381 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:03.587541 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 17:58:03.587811 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":129,"threads_started":1}
I20260929 17:58:03.590386 12300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1128ed8b21d44107bfc2b87ca453b572: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:03.591339 12300 log.cc:824] T 00000000000000000000000000000000 P 1128ed8b21d44107bfc2b87ca453b572: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:03.593597 12300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1128ed8b21d44107bfc2b87ca453b572: No bootstrap required, opened a new log
I20260929 17:58:03.597654 12300 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1128ed8b21d44107bfc2b87ca453b572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1128ed8b21d44107bfc2b87ca453b572" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 41979 } }
I20260929 17:58:03.597950 12300 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1128ed8b21d44107bfc2b87ca453b572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:03.597987 12300 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1128ed8b21d44107bfc2b87ca453b572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1128ed8b21d44107bfc2b87ca453b572, State: Initialized, Role: FOLLOWER
I20260929 17:58:03.598531 12300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1128ed8b21d44107bfc2b87ca453b572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1128ed8b21d44107bfc2b87ca453b572" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 41979 } }
I20260929 17:58:03.598701 12300 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1128ed8b21d44107bfc2b87ca453b572 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:58:03.598848 12300 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1128ed8b21d44107bfc2b87ca453b572 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:58:03.599002 12300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1128ed8b21d44107bfc2b87ca453b572 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:03.600236 12300 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1128ed8b21d44107bfc2b87ca453b572 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1128ed8b21d44107bfc2b87ca453b572" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 41979 } }
I20260929 17:58:03.600667 12300 leader_election.cc:302] T 00000000000000000000000000000000 P 1128ed8b21d44107bfc2b87ca453b572 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1128ed8b21d44107bfc2b87ca453b572; no voters:
I20260929 17:58:03.600986 12300 leader_election.cc:288] T 00000000000000000000000000000000 P 1128ed8b21d44107bfc2b87ca453b572 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:58:03.601137 12305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1128ed8b21d44107bfc2b87ca453b572 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:58:03.601430 12305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1128ed8b21d44107bfc2b87ca453b572 [term 1 LEADER]: Becoming Leader. State: Replica: 1128ed8b21d44107bfc2b87ca453b572, State: Running, Role: LEADER
I20260929 17:58:03.601819 12305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1128ed8b21d44107bfc2b87ca453b572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1128ed8b21d44107bfc2b87ca453b572" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 41979 } }
I20260929 17:58:03.602516 12300 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1128ed8b21d44107bfc2b87ca453b572 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:58:03.603756 12307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1128ed8b21d44107bfc2b87ca453b572 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1128ed8b21d44107bfc2b87ca453b572. Latest consensus state: current_term: 1 leader_uuid: "1128ed8b21d44107bfc2b87ca453b572" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1128ed8b21d44107bfc2b87ca453b572" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 41979 } } }
I20260929 17:58:03.603909 12307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1128ed8b21d44107bfc2b87ca453b572 [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:03.604235 12306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1128ed8b21d44107bfc2b87ca453b572 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1128ed8b21d44107bfc2b87ca453b572" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1128ed8b21d44107bfc2b87ca453b572" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 41979 } } }
I20260929 17:58:03.604344 12306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1128ed8b21d44107bfc2b87ca453b572 [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:03.604331 12314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:58:03.607865 12314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:58:03.613757 12314 catalog_manager.cc:1347] Generated new cluster ID: 930677ec0c2f44adad2dc344381a7afa
I20260929 17:58:03.613853 12314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:58:03.629899 12314 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:58:03.630380 12314 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:58:03.637955 12314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1128ed8b21d44107bfc2b87ca453b572: Generated new TSK 0
I20260929 17:58:03.638937 12314 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:58:03.649801 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:0
--local_ip_for_outbound_sockets=127.7.118.1
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:41979
--builtin_ntp_servers=127.7.118.20:46463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 17:58:03.770473 12324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:03.770797 12324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:03.770840 12324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:03.775259 12324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:03.775343 12324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:03.775410 12324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 17:58:03.780587 12324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:46463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:41979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:03.781821 12324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:03.783422 12324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:03.795754 12330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:03.795765 12324 server_base.cc:1034] running on GCE node
W20260929 17:58:03.795750 12329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:03.795835 12332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:03.796656 12324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:03.797646 12324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:03.798880 12324 hybrid_clock.cc:648] HybridClock initialized: now 1790704683798847 us; error 49 us; skew 500 ppm
I20260929 17:58:03.801151 12324 webserver.cc:460] Webserver started at http://127.7.118.1:42769/ using document root <none> and password file <none>
I20260929 17:58:03.801874 12324 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:03.801983 12324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:03.802234 12324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:03.804761 12324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-0/data/instance:
uuid: "21ead6addfeb40dd9abacb3972e94071"
format_stamp: "Formatted at 2026-09-29 17:58:03 on dist-test-slave-r7ph"
server_key: "8c8c8681dd6834c9c67f56d6a3763367"
server_key_iv: "d9081b811f12076f7c1d39283074af44"
server_key_version: "encryptionkey@0"
I20260929 17:58:03.805388 12324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance:
uuid: "21ead6addfeb40dd9abacb3972e94071"
format_stamp: "Formatted at 2026-09-29 17:58:03 on dist-test-slave-r7ph"
server_key: "8c8c8681dd6834c9c67f56d6a3763367"
server_key_iv: "d9081b811f12076f7c1d39283074af44"
server_key_version: "encryptionkey@0"
I20260929 17:58:03.809522 12324 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260929 17:58:03.812492 12338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:03.814170 12324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:58:03.814471 12324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "21ead6addfeb40dd9abacb3972e94071"
format_stamp: "Formatted at 2026-09-29 17:58:03 on dist-test-slave-r7ph"
server_key: "8c8c8681dd6834c9c67f56d6a3763367"
server_key_iv: "d9081b811f12076f7c1d39283074af44"
server_key_version: "encryptionkey@0"
I20260929 17:58:03.814623 12324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:03.820308 12324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:03.821058 12324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:03.821277 12324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:03.821900 12324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:03.823006 12324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:03.823086 12324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:03.823163 12324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:03.823207 12324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:03.833844 12324 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:46823
I20260929 17:58:03.833870 12451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:46823 every 8 connection(s)
I20260929 17:58:03.834932 12324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 17:58:03.837118 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 12324
I20260929 17:58:03.837321 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance
I20260929 17:58:03.837735 7640 external_mini_cluster.cc:1428] Setting key a6a6acabf7421ee3ec557cfc895c194d
I20260929 17:58:03.839840 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:0
--local_ip_for_outbound_sockets=127.7.118.2
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:41979
--builtin_ntp_servers=127.7.118.20:46463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 17:58:03.842164 12303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:03.837123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:34761 (local address 127.7.118.62:41979)
0929 17:58:03.837326 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:03.837333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:03.838716 (+ 1383us) server_negotiation.cc:407] Connection header received
0929 17:58:03.840212 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:03.840216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:03.840263 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:03.840362 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:03.841649 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:03.841653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:03.841656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:58:03.841669 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:03.841693 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:58:03.841709 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:03.841813 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:03.841978 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 17:58:03.842027 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 17:58:03.843145 12454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:03.837639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:41979 (local address 127.7.118.1:34761)
0929 17:58:03.838547 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:03.838608 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:03.839882 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:03.840571 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:03.840578 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:03.840680 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:03.841347 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:58:03.841348 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:03.841406 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:58:03.841484 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:03.841813 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:03.841826 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:03.841834 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 17:58:03.842027 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 17:58:03.842322 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":199,"threads_started":1}
I20260929 17:58:03.844669 12452 heartbeater.cc:347] Connected to a master server at 127.7.118.62:41979
I20260929 17:58:03.844973 12452 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:03.845440 12452 heartbeater.cc:511] Master 127.7.118.62:41979 requested a full tablet report, sending...
I20260929 17:58:03.847419 12264 ts_manager.cc:194] Registered new tserver with Master: 21ead6addfeb40dd9abacb3972e94071 (127.7.118.1:46823)
W20260929 17:58:03.960085 12455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:03.960420 12455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:03.960465 12455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:03.964843 12455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:03.964917 12455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:03.964979 12455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 17:58:03.970329 12455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:46463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:41979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:03.971483 12455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:03.972872 12455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:58:03.983745 12455 server_base.cc:1034] running on GCE node
W20260929 17:58:03.983798 12461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:03.983771 12463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:03.983712 12460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:03.984565 12455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:03.985319 12455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:03.986560 12455 hybrid_clock.cc:648] HybridClock initialized: now 1790704683986538 us; error 40 us; skew 500 ppm
I20260929 17:58:03.988895 12455 webserver.cc:460] Webserver started at http://127.7.118.2:35039/ using document root <none> and password file <none>
I20260929 17:58:03.989598 12455 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:03.989709 12455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:03.989971 12455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:03.992323 12455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-1/data/instance:
uuid: "541c0bb77e784188983052d74b0b0580"
format_stamp: "Formatted at 2026-09-29 17:58:03 on dist-test-slave-r7ph"
server_key: "ffb7abfc229b0ee5c45b9e556a3ca773"
server_key_iv: "92d81b9a41a766755272209be564be74"
server_key_version: "encryptionkey@0"
I20260929 17:58:03.992978 12455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance:
uuid: "541c0bb77e784188983052d74b0b0580"
format_stamp: "Formatted at 2026-09-29 17:58:03 on dist-test-slave-r7ph"
server_key: "ffb7abfc229b0ee5c45b9e556a3ca773"
server_key_iv: "92d81b9a41a766755272209be564be74"
server_key_version: "encryptionkey@0"
I20260929 17:58:03.997334 12455 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 17:58:04.000591 12469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:04.002468 12455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:58:04.002616 12455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "541c0bb77e784188983052d74b0b0580"
format_stamp: "Formatted at 2026-09-29 17:58:03 on dist-test-slave-r7ph"
server_key: "ffb7abfc229b0ee5c45b9e556a3ca773"
server_key_iv: "92d81b9a41a766755272209be564be74"
server_key_version: "encryptionkey@0"
I20260929 17:58:04.002777 12455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:04.008128 12455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:04.008812 12455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:04.009035 12455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:04.009632 12455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:04.010692 12455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:04.010740 12455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:04.010779 12455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:04.010795 12455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:04.020642 12455 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:33591
I20260929 17:58:04.020686 12582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:33591 every 8 connection(s)
I20260929 17:58:04.021634 12455 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 17:58:04.027001 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 12455
I20260929 17:58:04.027143 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance
I20260929 17:58:04.027465 7640 external_mini_cluster.cc:1428] Setting key d59d81d608b124cfee71b47f40168d59
I20260929 17:58:04.029275 12303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:04.023783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:34541 (local address 127.7.118.62:41979)
0929 17:58:04.023958 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:04.023962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:04.025255 (+ 1293us) server_negotiation.cc:407] Connection header received
0929 17:58:04.027060 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:04.027068 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:04.027128 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:04.027311 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:04.028673 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:04.028677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:04.028680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:58:04.028700 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:04.028722 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:58:04.028736 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:04.028854 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:04.029044 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 17:58:04.029097 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 17:58:04.029605 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:0
--local_ip_for_outbound_sockets=127.7.118.3
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:41979
--builtin_ntp_servers=127.7.118.20:46463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 17:58:04.030514 12585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:04.024220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:41979 (local address 127.7.118.2:34541)
0929 17:58:04.025063 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:04.025115 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:04.026758 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:04.027445 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:04.027454 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:04.027615 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:04.028352 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:58:04.028353 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:04.028408 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:58:04.028508 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:04.028868 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:04.028877 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:04.028883 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:58:04.029103 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 17:58:04.029422 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":135,"threads_started":1}
I20260929 17:58:04.031936 12583 heartbeater.cc:347] Connected to a master server at 127.7.118.62:41979
I20260929 17:58:04.032205 12583 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:04.032698 12583 heartbeater.cc:511] Master 127.7.118.62:41979 requested a full tablet report, sending...
I20260929 17:58:04.034060 12264 ts_manager.cc:194] Registered new tserver with Master: 541c0bb77e784188983052d74b0b0580 (127.7.118.2:33591)
W20260929 17:58:04.149493 12586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:04.149886 12586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:04.149932 12586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:04.154422 12586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:04.154510 12586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:04.154572 12586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 17:58:04.159845 12586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:46463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:41979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:04.161017 12586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:04.162869 12586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:04.174897 12591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:04.174917 12594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:04.175104 12586 server_base.cc:1034] running on GCE node
W20260929 17:58:04.174897 12592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:04.175691 12586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:04.176453 12586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:04.177654 12586 hybrid_clock.cc:648] HybridClock initialized: now 1790704684177611 us; error 59 us; skew 500 ppm
I20260929 17:58:04.180172 12586 webserver.cc:460] Webserver started at http://127.7.118.3:34235/ using document root <none> and password file <none>
I20260929 17:58:04.180889 12586 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:04.180963 12586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:04.181190 12586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:04.183588 12586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-2/data/instance:
uuid: "963261a43a714534a8a10ddd0ec1275c"
format_stamp: "Formatted at 2026-09-29 17:58:04 on dist-test-slave-r7ph"
server_key: "a1f643e6a67f5cb0ecf43828b358c2be"
server_key_iv: "268b68bfc83c675f74f5435a230e1b5b"
server_key_version: "encryptionkey@0"
I20260929 17:58:04.184240 12586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance:
uuid: "963261a43a714534a8a10ddd0ec1275c"
format_stamp: "Formatted at 2026-09-29 17:58:04 on dist-test-slave-r7ph"
server_key: "a1f643e6a67f5cb0ecf43828b358c2be"
server_key_iv: "268b68bfc83c675f74f5435a230e1b5b"
server_key_version: "encryptionkey@0"
I20260929 17:58:04.188786 12586 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:58:04.191952 12600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:04.193728 12586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 17:58:04.193934 12586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "963261a43a714534a8a10ddd0ec1275c"
format_stamp: "Formatted at 2026-09-29 17:58:04 on dist-test-slave-r7ph"
server_key: "a1f643e6a67f5cb0ecf43828b358c2be"
server_key_iv: "268b68bfc83c675f74f5435a230e1b5b"
server_key_version: "encryptionkey@0"
I20260929 17:58:04.194094 12586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:04.199664 12586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:04.200431 12586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:04.200640 12586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:04.201273 12586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:04.202446 12586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:04.202498 12586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:04.202541 12586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:04.202593 12586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:04.212594 12586 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:46035
I20260929 17:58:04.212682 12713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:46035 every 8 connection(s)
I20260929 17:58:04.213642 12586 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 17:58:04.218928 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 12586
I20260929 17:58:04.219075 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance
I20260929 17:58:04.219502 7640 external_mini_cluster.cc:1428] Setting key 8bdc69cc8c55769ac6de12029972e894
I20260929 17:58:04.221576 12303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:04.216169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:51443 (local address 127.7.118.62:41979)
0929 17:58:04.216392 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:04.216396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:04.217679 (+ 1283us) server_negotiation.cc:407] Connection header received
0929 17:58:04.219316 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:04.219320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:04.219382 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:04.219470 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:04.220974 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:04.220980 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:04.220984 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:58:04.221000 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:04.221028 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:58:04.221046 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:04.221120 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:04.221341 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 17:58:04.221441 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 17:58:04.222857 12716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:04.216689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:41979 (local address 127.7.118.3:51443)
0929 17:58:04.217484 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:04.217533 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:04.218996 (+ 1463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:04.219687 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:04.219696 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:04.219808 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:04.220597 (+ 789us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:58:04.220599 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:04.220670 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:58:04.220787 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:04.221145 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:04.221156 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:04.221164 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 17:58:04.221359 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 17:58:04.221628 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":141,"threads_started":1}
I20260929 17:58:04.224157 12714 heartbeater.cc:347] Connected to a master server at 127.7.118.62:41979
I20260929 17:58:04.224411 12714 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:04.224836 12714 heartbeater.cc:511] Master 127.7.118.62:41979 requested a full tablet report, sending...
I20260929 17:58:04.225999 12264 ts_manager.cc:194] Registered new tserver with Master: 963261a43a714534a8a10ddd0ec1275c (127.7.118.3:46035)
I20260929 17:58:04.234894 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:58:04.238977 12303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:04.237031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49768 (local address 127.7.118.62:41979)
0929 17:58:04.237196 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:04.237203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:04.237387 (+ 184us) server_negotiation.cc:407] Connection header received
0929 17:58:04.237485 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:04.237487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:04.237527 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:04.237599 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:04.238480 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:04.238483 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:04.238486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:58:04.238498 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:04.238536 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:58:04.238546 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:04.238644 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:04.238692 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 17:58:04.238729 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 17:58:04.242939 12264 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:58:04.246140 12264 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:58:04.259763 12726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:04.256020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:46035 (local address 127.0.0.1:33926)
0929 17:58:04.256490 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:04.256550 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:04.256656 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:04.257737 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:04.257744 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:04.257832 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:04.258703 (+ 871us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:58:04.258734 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:04.258757 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:58:04.259059 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:04.259428 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:04.259431 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:04.259435 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 17:58:04.259527 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 17:58:04.259599 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":96,"threads_started":1}
I20260929 17:58:04.259810 12730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:04.256979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45854 (local address 127.7.118.1:46823)
0929 17:58:04.257845 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:04.257860 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:04.257885 (+ 25us) server_negotiation.cc:407] Connection header received
0929 17:58:04.257936 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:04.257941 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:04.258114 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:04.258339 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:04.259076 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:04.259083 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:04.259097 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:58:04.259109 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:04.259157 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:58:04.259192 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:04.259327 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:04.259532 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 17:58:04.259613 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":89,"threads_started":1}
I20260929 17:58:04.260262 12727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:04.256149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33926 (local address 127.7.118.3:46035)
0929 17:58:04.257215 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:04.257229 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:04.257260 (+ 31us) server_negotiation.cc:407] Connection header received
0929 17:58:04.257317 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:04.257324 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:04.257562 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:04.257715 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:04.259180 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:04.259186 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:04.259198 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:58:04.259211 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:04.259283 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:58:04.259320 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:04.259441 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:04.259978 (+ 537us) server_negotiation.cc:299] Negotiation successful
0929 17:58:04.260064 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":958,"thread_start_us":199,"threads_started":1}
I20260929 17:58:04.260578 12728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:04.256083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:33591 (local address 127.0.0.1:46346)
0929 17:58:04.257288 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:04.257301 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:04.257414 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:04.258588 (+ 1174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:04.258593 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:04.258614 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:04.259546 (+ 932us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:58:04.259548 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:04.259558 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:58:04.259580 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:04.260215 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:04.260218 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:04.260221 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 17:58:04.260303 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 17:58:04.260437 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1134,"thread_start_us":126,"threads_started":1}
I20260929 17:58:04.260777 12725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:04.256834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:46823 (local address 127.0.0.1:45854)
0929 17:58:04.257503 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:04.257518 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:04.257584 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:04.258317 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:04.258322 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:04.258340 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:04.258823 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:58:04.258826 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:04.258836 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:58:04.258932 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:04.259297 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:04.259307 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:04.259314 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 17:58:04.259434 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 17:58:04.260640 (+ 1206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":105,"threads_started":1}
I20260929 17:58:04.261255 12729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:04.256875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46346 (local address 127.7.118.2:33591)
0929 17:58:04.257674 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:04.257686 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:04.257711 (+ 25us) server_negotiation.cc:407] Connection header received
0929 17:58:04.257761 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:04.257765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:04.257986 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:04.258863 (+ 877us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:04.259707 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:04.259712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:04.259722 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:58:04.259734 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:04.259774 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:58:04.259807 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:04.260163 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:04.261023 (+ 860us) server_negotiation.cc:299] Negotiation successful
0929 17:58:04.261096 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":117,"threads_started":1}
I20260929 17:58:04.263854 12517 tablet_service.cc:1467] Processing CreateTablet for tablet a5a9818db53c45ac9e65a80af0c34b6d (DEFAULT_TABLE table=test-table [id=a8cb8556aaae49fc90b4d4fe1c9b67df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:58:04.263864 12648 tablet_service.cc:1467] Processing CreateTablet for tablet a5a9818db53c45ac9e65a80af0c34b6d (DEFAULT_TABLE table=test-table [id=a8cb8556aaae49fc90b4d4fe1c9b67df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:58:04.263868 12386 tablet_service.cc:1467] Processing CreateTablet for tablet a5a9818db53c45ac9e65a80af0c34b6d (DEFAULT_TABLE table=test-table [id=a8cb8556aaae49fc90b4d4fe1c9b67df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:58:04.265065 12648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5a9818db53c45ac9e65a80af0c34b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:04.265084 12517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5a9818db53c45ac9e65a80af0c34b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:04.265472 12386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5a9818db53c45ac9e65a80af0c34b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:04.271939 12731 tablet_bootstrap.cc:492] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580: Bootstrap starting.
I20260929 17:58:04.273819 12732 tablet_bootstrap.cc:492] T a5a9818db53c45ac9e65a80af0c34b6d P 963261a43a714534a8a10ddd0ec1275c: Bootstrap starting.
I20260929 17:58:04.274920 12731 tablet_bootstrap.cc:654] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:04.275074 12733 tablet_bootstrap.cc:492] T a5a9818db53c45ac9e65a80af0c34b6d P 21ead6addfeb40dd9abacb3972e94071: Bootstrap starting.
I20260929 17:58:04.276057 12731 log.cc:824] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:04.276684 12732 tablet_bootstrap.cc:654] T a5a9818db53c45ac9e65a80af0c34b6d P 963261a43a714534a8a10ddd0ec1275c: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:04.277601 12732 log.cc:824] T a5a9818db53c45ac9e65a80af0c34b6d P 963261a43a714534a8a10ddd0ec1275c: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:04.277828 12733 tablet_bootstrap.cc:654] T a5a9818db53c45ac9e65a80af0c34b6d P 21ead6addfeb40dd9abacb3972e94071: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:04.278774 12733 log.cc:824] T a5a9818db53c45ac9e65a80af0c34b6d P 21ead6addfeb40dd9abacb3972e94071: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:04.278903 12731 tablet_bootstrap.cc:492] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580: No bootstrap required, opened a new log
I20260929 17:58:04.279124 12731 ts_tablet_manager.cc:1397] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 17:58:04.279748 12732 tablet_bootstrap.cc:492] T a5a9818db53c45ac9e65a80af0c34b6d P 963261a43a714534a8a10ddd0ec1275c: No bootstrap required, opened a new log
I20260929 17:58:04.279973 12732 ts_tablet_manager.cc:1397] T a5a9818db53c45ac9e65a80af0c34b6d P 963261a43a714534a8a10ddd0ec1275c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 17:58:04.280984 12733 tablet_bootstrap.cc:492] T a5a9818db53c45ac9e65a80af0c34b6d P 21ead6addfeb40dd9abacb3972e94071: No bootstrap required, opened a new log
I20260929 17:58:04.281180 12733 ts_tablet_manager.cc:1397] T a5a9818db53c45ac9e65a80af0c34b6d P 21ead6addfeb40dd9abacb3972e94071: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 17:58:04.284251 12732 raft_consensus.cc:348] T a5a9818db53c45ac9e65a80af0c34b6d P 963261a43a714534a8a10ddd0ec1275c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963261a43a714534a8a10ddd0ec1275c" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 46035 } } peers { permanent_uuid: "541c0bb77e784188983052d74b0b0580" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 33591 } } peers { permanent_uuid: "21ead6addfeb40dd9abacb3972e94071" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 46823 } }
I20260929 17:58:04.284593 12732 raft_consensus.cc:374] T a5a9818db53c45ac9e65a80af0c34b6d P 963261a43a714534a8a10ddd0ec1275c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:04.284690 12732 raft_consensus.cc:729] T a5a9818db53c45ac9e65a80af0c34b6d P 963261a43a714534a8a10ddd0ec1275c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 963261a43a714534a8a10ddd0ec1275c, State: Initialized, Role: FOLLOWER
I20260929 17:58:04.284889 12731 raft_consensus.cc:348] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963261a43a714534a8a10ddd0ec1275c" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 46035 } } peers { permanent_uuid: "541c0bb77e784188983052d74b0b0580" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 33591 } } peers { permanent_uuid: "21ead6addfeb40dd9abacb3972e94071" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 46823 } }
I20260929 17:58:04.285146 12731 raft_consensus.cc:374] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:04.285225 12731 raft_consensus.cc:729] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 541c0bb77e784188983052d74b0b0580, State: Initialized, Role: FOLLOWER
I20260929 17:58:04.285224 12732 consensus_queue.cc:260] T a5a9818db53c45ac9e65a80af0c34b6d P 963261a43a714534a8a10ddd0ec1275c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963261a43a714534a8a10ddd0ec1275c" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 46035 } } peers { permanent_uuid: "541c0bb77e784188983052d74b0b0580" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 33591 } } peers { permanent_uuid: "21ead6addfeb40dd9abacb3972e94071" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 46823 } }
I20260929 17:58:04.285075 12733 raft_consensus.cc:348] T a5a9818db53c45ac9e65a80af0c34b6d P 21ead6addfeb40dd9abacb3972e94071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963261a43a714534a8a10ddd0ec1275c" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 46035 } } peers { permanent_uuid: "541c0bb77e784188983052d74b0b0580" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 33591 } } peers { permanent_uuid: "21ead6addfeb40dd9abacb3972e94071" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 46823 } }
I20260929 17:58:04.285378 12733 raft_consensus.cc:374] T a5a9818db53c45ac9e65a80af0c34b6d P 21ead6addfeb40dd9abacb3972e94071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:04.285455 12733 raft_consensus.cc:729] T a5a9818db53c45ac9e65a80af0c34b6d P 21ead6addfeb40dd9abacb3972e94071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21ead6addfeb40dd9abacb3972e94071, State: Initialized, Role: FOLLOWER
I20260929 17:58:04.285701 12731 consensus_queue.cc:260] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963261a43a714534a8a10ddd0ec1275c" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 46035 } } peers { permanent_uuid: "541c0bb77e784188983052d74b0b0580" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 33591 } } peers { permanent_uuid: "21ead6addfeb40dd9abacb3972e94071" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 46823 } }
I20260929 17:58:04.285943 12733 consensus_queue.cc:260] T a5a9818db53c45ac9e65a80af0c34b6d P 21ead6addfeb40dd9abacb3972e94071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963261a43a714534a8a10ddd0ec1275c" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 46035 } } peers { permanent_uuid: "541c0bb77e784188983052d74b0b0580" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 33591 } } peers { permanent_uuid: "21ead6addfeb40dd9abacb3972e94071" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 46823 } }
I20260929 17:58:04.286087 12714 heartbeater.cc:503] Master 127.7.118.62:41979 was elected leader, sending a full tablet report...
I20260929 17:58:04.286458 12583 heartbeater.cc:503] Master 127.7.118.62:41979 was elected leader, sending a full tablet report...
I20260929 17:58:04.286871 12731 ts_tablet_manager.cc:1428] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260929 17:58:04.287638 12732 ts_tablet_manager.cc:1428] T a5a9818db53c45ac9e65a80af0c34b6d P 963261a43a714534a8a10ddd0ec1275c: Time spent starting tablet: real 0.008s user 0.001s sys 0.006s
I20260929 17:58:04.288033 12452 heartbeater.cc:503] Master 127.7.118.62:41979 was elected leader, sending a full tablet report...
I20260929 17:58:04.288938 12733 ts_tablet_manager.cc:1428] T a5a9818db53c45ac9e65a80af0c34b6d P 21ead6addfeb40dd9abacb3972e94071: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260929 17:58:04.336241 12453 tablet.cc:2369] T a5a9818db53c45ac9e65a80af0c34b6d P 21ead6addfeb40dd9abacb3972e94071: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:58:04.405453 12738 raft_consensus.cc:482] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:58:04.405706 12738 raft_consensus.cc:504] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963261a43a714534a8a10ddd0ec1275c" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 46035 } } peers { permanent_uuid: "541c0bb77e784188983052d74b0b0580" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 33591 } } peers { permanent_uuid: "21ead6addfeb40dd9abacb3972e94071" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 46823 } }
I20260929 17:58:04.406946 12738 leader_election.cc:288] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 963261a43a714534a8a10ddd0ec1275c (127.7.118.3:46035), 21ead6addfeb40dd9abacb3972e94071 (127.7.118.1:46823)
I20260929 17:58:04.409408 12730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:04.407814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:45735 (local address 127.7.118.1:46823)
0929 17:58:04.407959 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:04.407964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:04.408148 (+ 184us) server_negotiation.cc:407] Connection header received
0929 17:58:04.408238 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:04.408241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:04.408280 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:04.408371 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:04.408997 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:04.409016 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:04.409019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:58:04.409030 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:04.409049 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:58:04.409058 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:04.409129 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:04.409240 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 17:58:04.409281 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 17:58:04.409444 12585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:04.407354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:46035 (local address 127.7.118.2:60217)
0929 17:58:04.407523 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:04.407546 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:04.407657 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:04.408017 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:04.408020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:04.408034 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:04.408627 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:58:04.408630 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:04.408644 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:58:04.408676 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:04.409166 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:04.409169 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:04.409171 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 17:58:04.409277 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 17:58:04.409315 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 17:58:04.409444 12727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:04.407452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:60217 (local address 127.7.118.3:46035)
0929 17:58:04.407612 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:04.407618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:04.407636 (+ 18us) server_negotiation.cc:407] Connection header received
0929 17:58:04.407773 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:04.407776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:04.407822 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:04.407913 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:04.408816 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:04.408822 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:04.408824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:58:04.408846 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:04.408866 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:58:04.408876 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:04.408974 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:04.409284 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 17:58:04.409326 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 17:58:04.409487 12740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:04.407667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:46823 (local address 127.7.118.2:45735)
0929 17:58:04.408040 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:04.408051 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:04.408151 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:04.408390 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:04.408393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:04.408406 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:04.408839 (+ 433us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:58:04.408842 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:04.408852 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:58:04.408877 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:04.409128 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:04.409130 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:04.409133 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 17:58:04.409227 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 17:58:04.409265 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":140,"threads_started":1}
I20260929 17:58:04.410378 12406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a9818db53c45ac9e65a80af0c34b6d" candidate_uuid: "541c0bb77e784188983052d74b0b0580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ead6addfeb40dd9abacb3972e94071" is_pre_election: true
I20260929 17:58:04.410387 12668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a9818db53c45ac9e65a80af0c34b6d" candidate_uuid: "541c0bb77e784188983052d74b0b0580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "963261a43a714534a8a10ddd0ec1275c" is_pre_election: true
I20260929 17:58:04.410699 12406 raft_consensus.cc:2454] T a5a9818db53c45ac9e65a80af0c34b6d P 21ead6addfeb40dd9abacb3972e94071 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 541c0bb77e784188983052d74b0b0580 in term 0.
I20260929 17:58:04.410722 12668 raft_consensus.cc:2454] T a5a9818db53c45ac9e65a80af0c34b6d P 963261a43a714534a8a10ddd0ec1275c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 541c0bb77e784188983052d74b0b0580 in term 0.
I20260929 17:58:04.411206 12473 leader_election.cc:302] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21ead6addfeb40dd9abacb3972e94071, 541c0bb77e784188983052d74b0b0580; no voters:
I20260929 17:58:04.411518 12738 raft_consensus.cc:2781] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:58:04.411618 12738 raft_consensus.cc:482] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:58:04.411700 12738 raft_consensus.cc:3037] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:04.412945 12738 raft_consensus.cc:504] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963261a43a714534a8a10ddd0ec1275c" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 46035 } } peers { permanent_uuid: "541c0bb77e784188983052d74b0b0580" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 33591 } } peers { permanent_uuid: "21ead6addfeb40dd9abacb3972e94071" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 46823 } }
I20260929 17:58:04.413414 12738 leader_election.cc:288] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580 [CANDIDATE]: Term 1 election: Requested vote from peers 963261a43a714534a8a10ddd0ec1275c (127.7.118.3:46035), 21ead6addfeb40dd9abacb3972e94071 (127.7.118.1:46823)
I20260929 17:58:04.413793 12668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a9818db53c45ac9e65a80af0c34b6d" candidate_uuid: "541c0bb77e784188983052d74b0b0580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "963261a43a714534a8a10ddd0ec1275c"
I20260929 17:58:04.413857 12406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a9818db53c45ac9e65a80af0c34b6d" candidate_uuid: "541c0bb77e784188983052d74b0b0580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ead6addfeb40dd9abacb3972e94071"
I20260929 17:58:04.413954 12668 raft_consensus.cc:3037] T a5a9818db53c45ac9e65a80af0c34b6d P 963261a43a714534a8a10ddd0ec1275c [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:04.413995 12406 raft_consensus.cc:3037] T a5a9818db53c45ac9e65a80af0c34b6d P 21ead6addfeb40dd9abacb3972e94071 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:04.415097 12406 raft_consensus.cc:2454] T a5a9818db53c45ac9e65a80af0c34b6d P 21ead6addfeb40dd9abacb3972e94071 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 541c0bb77e784188983052d74b0b0580 in term 1.
I20260929 17:58:04.415109 12668 raft_consensus.cc:2454] T a5a9818db53c45ac9e65a80af0c34b6d P 963261a43a714534a8a10ddd0ec1275c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 541c0bb77e784188983052d74b0b0580 in term 1.
I20260929 17:58:04.415446 12473 leader_election.cc:302] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21ead6addfeb40dd9abacb3972e94071, 541c0bb77e784188983052d74b0b0580; no voters:
I20260929 17:58:04.415632 12738 raft_consensus.cc:2781] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:58:04.415895 12738 raft_consensus.cc:686] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580 [term 1 LEADER]: Becoming Leader. State: Replica: 541c0bb77e784188983052d74b0b0580, State: Running, Role: LEADER
I20260929 17:58:04.416273 12738 consensus_queue.cc:237] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963261a43a714534a8a10ddd0ec1275c" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 46035 } } peers { permanent_uuid: "541c0bb77e784188983052d74b0b0580" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 33591 } } peers { permanent_uuid: "21ead6addfeb40dd9abacb3972e94071" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 46823 } }
I20260929 17:58:04.419838 12260 catalog_manager.cc:5495] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580 reported cstate change: term changed from 0 to 1, leader changed from <none> to 541c0bb77e784188983052d74b0b0580 (127.7.118.2). New cstate: current_term: 1 leader_uuid: "541c0bb77e784188983052d74b0b0580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963261a43a714534a8a10ddd0ec1275c" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 46035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "541c0bb77e784188983052d74b0b0580" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 33591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21ead6addfeb40dd9abacb3972e94071" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 46823 } health_report { overall_health: UNKNOWN } } }
I20260929 17:58:04.457269 12729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:04.455474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46354 (local address 127.7.118.2:33591)
0929 17:58:04.455651 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:04.455656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:04.455673 (+ 17us) server_negotiation.cc:407] Connection header received
0929 17:58:04.455800 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:04.455802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:04.455886 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:04.455984 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:04.456691 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:04.456694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:04.456696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:58:04.456707 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:04.456729 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:58:04.456737 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:04.456827 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:04.456967 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 17:58:04.457037 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 17:58:04.464118 12668 raft_consensus.cc:1261] T a5a9818db53c45ac9e65a80af0c34b6d P 963261a43a714534a8a10ddd0ec1275c [term 1 FOLLOWER]: Refusing update from remote peer 541c0bb77e784188983052d74b0b0580: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:58:04.464118 12406 raft_consensus.cc:1261] T a5a9818db53c45ac9e65a80af0c34b6d P 21ead6addfeb40dd9abacb3972e94071 [term 1 FOLLOWER]: Refusing update from remote peer 541c0bb77e784188983052d74b0b0580: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 17:58:04.464733 12715 tablet.cc:2369] T a5a9818db53c45ac9e65a80af0c34b6d P 963261a43a714534a8a10ddd0ec1275c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:58:04.464977 12741 consensus_queue.cc:1035] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21ead6addfeb40dd9abacb3972e94071" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:58:04.465327 12738 consensus_queue.cc:1035] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "963261a43a714534a8a10ddd0ec1275c" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 46035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:58:04.475581 12746 mvcc.cc:205] Tried to move back new op lower bound from 7334726387555901440 to 7334726387371945984. Current Snapshot: MvccSnapshot[applied={T|T < 7334726387555901440}]
I20260929 17:58:04.475831 12745 mvcc.cc:205] Tried to move back new op lower bound from 7334726387555901440 to 7334726387371945984. Current Snapshot: MvccSnapshot[applied={T|T < 7334726387555901440}]
I20260929 17:58:04.483772 12260 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:58:04.483952 12260 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:58:04.486766 12260 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1128ed8b21d44107bfc2b87ca453b572: Sending DeleteTablet for 3 replicas of tablet a5a9818db53c45ac9e65a80af0c34b6d
I20260929 17:58:04.487780 12648 tablet_service.cc:1514] Processing DeleteTablet for tablet a5a9818db53c45ac9e65a80af0c34b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:58:04 UTC) from {username='slave'} at 127.0.0.1:33926
I20260929 17:58:04.487778 12517 tablet_service.cc:1514] Processing DeleteTablet for tablet a5a9818db53c45ac9e65a80af0c34b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:58:04 UTC) from {username='slave'} at 127.0.0.1:46346
I20260929 17:58:04.487854 12386 tablet_service.cc:1514] Processing DeleteTablet for tablet a5a9818db53c45ac9e65a80af0c34b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:58:04 UTC) from {username='slave'} at 127.0.0.1:45854
I20260929 17:58:04.488389 12754 tablet_replica.cc:330] T a5a9818db53c45ac9e65a80af0c34b6d P 963261a43a714534a8a10ddd0ec1275c: stopping tablet replica
I20260929 17:58:04.488464 12755 tablet_replica.cc:330] T a5a9818db53c45ac9e65a80af0c34b6d P 21ead6addfeb40dd9abacb3972e94071: stopping tablet replica
I20260929 17:58:04.488503 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 12324
I20260929 17:58:04.488440 12756 tablet_replica.cc:330] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580: stopping tablet replica
I20260929 17:58:04.488873 12755 raft_consensus.cc:2229] T a5a9818db53c45ac9e65a80af0c34b6d P 21ead6addfeb40dd9abacb3972e94071 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:58:04.488847 12754 raft_consensus.cc:2229] T a5a9818db53c45ac9e65a80af0c34b6d P 963261a43a714534a8a10ddd0ec1275c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:58:04.489151 12756 raft_consensus.cc:2229] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:58:04.489449 12754 raft_consensus.cc:2258] T a5a9818db53c45ac9e65a80af0c34b6d P 963261a43a714534a8a10ddd0ec1275c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:58:04.489641 12756 raft_consensus.cc:2258] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:58:04.492960 12756 ts_tablet_manager.cc:1905] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:58:04.492996 12754 ts_tablet_manager.cc:1905] T a5a9818db53c45ac9e65a80af0c34b6d P 963261a43a714534a8a10ddd0ec1275c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:58:04.495949 12754 ts_tablet_manager.cc:1918] T a5a9818db53c45ac9e65a80af0c34b6d P 963261a43a714534a8a10ddd0ec1275c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:58:04.496116 12754 log.cc:1196] T a5a9818db53c45ac9e65a80af0c34b6d P 963261a43a714534a8a10ddd0ec1275c: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-2/wal/wals/a5a9818db53c45ac9e65a80af0c34b6d
I20260929 17:58:04.496433 12754 ts_tablet_manager.cc:1939] T a5a9818db53c45ac9e65a80af0c34b6d P 963261a43a714534a8a10ddd0ec1275c: Deleting consensus metadata
I20260929 17:58:04.496937 12756 ts_tablet_manager.cc:1918] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:58:04.497068 12756 log.cc:1196] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704657566865-7640-0/minicluster-data/ts-1/wal/wals/a5a9818db53c45ac9e65a80af0c34b6d
I20260929 17:58:04.497557 12756 ts_tablet_manager.cc:1939] T a5a9818db53c45ac9e65a80af0c34b6d P 541c0bb77e784188983052d74b0b0580: Deleting consensus metadata
I20260929 17:58:04.497900 12251 catalog_manager.cc:4841] TS 963261a43a714534a8a10ddd0ec1275c (127.7.118.3:46035): tablet a5a9818db53c45ac9e65a80af0c34b6d (table test-table [id=a8cb8556aaae49fc90b4d4fe1c9b67df]) successfully deleted
I20260929 17:58:04.499185 12249 catalog_manager.cc:4841] TS 541c0bb77e784188983052d74b0b0580 (127.7.118.2:33591): tablet a5a9818db53c45ac9e65a80af0c34b6d (table test-table [id=a8cb8556aaae49fc90b4d4fe1c9b67df]) successfully deleted
W20260929 17:58:04.499651 12251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.118.1:46823 (error 108)
I20260929 17:58:04.499843 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 12455
I20260929 17:58:04.502564 12725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:04.502190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:46823 (local address 127.0.0.1:45866)
0929 17:58:04.502396 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:04.502477 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.1:46823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260929 17:58:04.502889 12251 catalog_manager.cc:4584] TS 21ead6addfeb40dd9abacb3972e94071 (127.7.118.1:46823): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a5a9818db53c45ac9e65a80af0c34b6d: Network error: Client connection negotiation failed: client connection to 127.7.118.1:46823: connect: Connection refused (error 111)
I20260929 17:58:04.506304 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 12586
I20260929 17:58:04.512162 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 12233
2026-09-29T17:58:04Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1149 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 17:58:04.534375 7640 mini_oidc.cc:152] Starting JWKS server
I20260929 17:58:04.535975 7640 webserver.cc:460] Webserver started at https://127.0.0.1:36037/ using document root <none> and password file <none>
I20260929 17:58:04.536123 7640 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 17:58:04.536392 7640 webserver.cc:460] Webserver started at http://127.0.0.1:34561/ using document root <none> and password file <none>
2026-09-29T17:58:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:58:04Z Disabled control of system clock
I20260929 17:58:04.546746 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:35743
--webserver_interface=127.7.118.62
--webserver_port=0
--builtin_ntp_servers=127.7.118.20:46799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:35743
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36037/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704657566865-7640-0/testchainca.cert with env {}
W20260929 17:58:04.665635 12763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:04.665956 12763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:04.666000 12763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:04.670569 12763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:58:04.670671 12763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:04.670691 12763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:04.670712 12763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:58:04.670732 12763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 17:58:04.670751 12763 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 17:58:04.670768 12763 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36037/jwks.json
I20260929 17:58:04.677450 12763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:46799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:35743
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:35743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36037/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704657566865-7640-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:04.678719 12763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:04.679790 12763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:04.689425 12769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:04.689425 12768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:04.689440 12771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:04.689920 12763 server_base.cc:1034] running on GCE node
I20260929 17:58:04.690608 12763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:04.691905 12763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:04.693117 12763 hybrid_clock.cc:648] HybridClock initialized: now 1790704684693101 us; error 44 us; skew 500 ppm
I20260929 17:58:04.695384 12763 webserver.cc:460] Webserver started at http://127.7.118.62:38423/ using document root <none> and password file <none>
I20260929 17:58:04.696033 12763 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:04.696134 12763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:04.696372 12763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:04.698618 12763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "b6ad2d83b569479f9aca09154c91d146"
format_stamp: "Formatted at 2026-09-29 17:58:04 on dist-test-slave-r7ph"
server_key: "94c974164d0fdd177334b2bca9f31de0"
server_key_iv: "ac5d3d19a0f400185bdab92f37038cff"
server_key_version: "encryptionkey@0"
I20260929 17:58:04.699234 12763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "b6ad2d83b569479f9aca09154c91d146"
format_stamp: "Formatted at 2026-09-29 17:58:04 on dist-test-slave-r7ph"
server_key: "94c974164d0fdd177334b2bca9f31de0"
server_key_iv: "ac5d3d19a0f400185bdab92f37038cff"
server_key_version: "encryptionkey@0"
I20260929 17:58:04.703372 12763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 17:58:04.706233 12777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:04.707911 12763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 17:58:04.708074 12763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "b6ad2d83b569479f9aca09154c91d146"
format_stamp: "Formatted at 2026-09-29 17:58:04 on dist-test-slave-r7ph"
server_key: "94c974164d0fdd177334b2bca9f31de0"
server_key_iv: "ac5d3d19a0f400185bdab92f37038cff"
server_key_version: "encryptionkey@0"
I20260929 17:58:04.708212 12763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:04.743819 12763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:04.744670 12763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:04.744923 12763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:04.753480 12829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:35743 every 8 connection(s)
I20260929 17:58:04.753504 12763 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:35743
I20260929 17:58:04.754721 12763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 17:58:04.758533 12830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:04.764360 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 12763
I20260929 17:58:04.764540 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 17:58:04.764860 7640 external_mini_cluster.cc:1428] Setting key bee35e3c6725f73d591e989683d937ca
I20260929 17:58:04.765168 12830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6ad2d83b569479f9aca09154c91d146: Bootstrap starting.
I20260929 17:58:04.768186 12830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6ad2d83b569479f9aca09154c91d146: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:04.769126 12830 log.cc:824] T 00000000000000000000000000000000 P b6ad2d83b569479f9aca09154c91d146: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:04.771771 12830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6ad2d83b569479f9aca09154c91d146: No bootstrap required, opened a new log
I20260929 17:58:04.776454 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:04.766581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45406 (local address 127.7.118.62:35743)
0929 17:58:04.767402 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:04.767418 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:04.767459 (+ 41us) server_negotiation.cc:407] Connection header received
0929 17:58:04.768183 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:04.768205 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:04.768768 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:04.769125 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:04.769607 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:04.770717 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:04.772030 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:04.772749 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:04.772894 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:04.773954 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:04.773959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:04.773975 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:58:04.774553 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:04.774619 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:58:04.774643 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:04.774734 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:04.774955 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 17:58:04.775524 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":157,"threads_started":1}
I20260929 17:58:04.776647 12830 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6ad2d83b569479f9aca09154c91d146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ad2d83b569479f9aca09154c91d146" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35743 } }
I20260929 17:58:04.776995 12830 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6ad2d83b569479f9aca09154c91d146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:04.777070 12830 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6ad2d83b569479f9aca09154c91d146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ad2d83b569479f9aca09154c91d146, State: Initialized, Role: FOLLOWER
I20260929 17:58:04.777673 12830 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6ad2d83b569479f9aca09154c91d146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ad2d83b569479f9aca09154c91d146" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35743 } }
I20260929 17:58:04.777977 12830 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6ad2d83b569479f9aca09154c91d146 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:58:04.778092 12830 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6ad2d83b569479f9aca09154c91d146 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:58:04.778520 12830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6ad2d83b569479f9aca09154c91d146 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:04.779839 12830 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6ad2d83b569479f9aca09154c91d146 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ad2d83b569479f9aca09154c91d146" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35743 } }
I20260929 17:58:04.780285 12830 leader_election.cc:302] T 00000000000000000000000000000000 P b6ad2d83b569479f9aca09154c91d146 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6ad2d83b569479f9aca09154c91d146; no voters:
I20260929 17:58:04.780570 7640 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:58:04.780707 12830 leader_election.cc:288] T 00000000000000000000000000000000 P b6ad2d83b569479f9aca09154c91d146 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:58:04.780874 12835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6ad2d83b569479f9aca09154c91d146 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:58:04.781142 12835 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6ad2d83b569479f9aca09154c91d146 [term 1 LEADER]: Becoming Leader. State: Replica: b6ad2d83b569479f9aca09154c91d146, State: Running, Role: LEADER
I20260929 17:58:04.781599 12835 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6ad2d83b569479f9aca09154c91d146 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ad2d83b569479f9aca09154c91d146" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35743 } }
I20260929 17:58:04.782202 12830 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6ad2d83b569479f9aca09154c91d146 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:58:04.783582 12836 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6ad2d83b569479f9aca09154c91d146 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6ad2d83b569479f9aca09154c91d146" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ad2d83b569479f9aca09154c91d146" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35743 } } }
I20260929 17:58:04.783744 12836 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6ad2d83b569479f9aca09154c91d146 [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:04.784114 12837 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6ad2d83b569479f9aca09154c91d146 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6ad2d83b569479f9aca09154c91d146. Latest consensus state: current_term: 1 leader_uuid: "b6ad2d83b569479f9aca09154c91d146" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ad2d83b569479f9aca09154c91d146" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35743 } } }
I20260929 17:58:04.784260 12837 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6ad2d83b569479f9aca09154c91d146 [sys.catalog]: This master's current role is: LEADER
W20260929 17:58:04.788424 12850 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b6ad2d83b569479f9aca09154c91d146: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:58:04.788529 12850 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:58:04.788637 12844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:58:04.789459 12844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:58:04.796891 12844 catalog_manager.cc:1347] Generated new cluster ID: 674a98f633b04f6d8938086298ff1cd2
I20260929 17:58:04.797006 12844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:58:04.810019 12844 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:58:04.811654 12844 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:58:04.821069 12844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6ad2d83b569479f9aca09154c91d146: Generated new TSK 0
I20260929 17:58:04.822053 12844 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 17:58:04.862142 12860 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 17:58:04.862320 12860 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 17:58:04.862820 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:04.837342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45418 (local address 127.7.118.62:35743)
0929 17:58:04.837510 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:04.837515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:04.837868 (+ 353us) server_negotiation.cc:407] Connection header received
0929 17:58:04.838028 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:04.838031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:04.838093 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:04.838191 (+ 98us) server_negotiation.cc:226] Negotiated authn=JWT
0929 17:58:04.838541 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:04.839159 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:04.840570 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:04.840870 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:04.841002 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:04.841491 (+ 489us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 17:58:04.862326 (+ 20835us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36037/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 17:58:04.862688 (+ 362us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45418: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36037/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 17:58:04.862946 12833 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45418: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36037/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 17:58:04.864409 12859 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:04.837274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:35743 (local address 127.0.0.1:45418)
0929 17:58:04.837667 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:04.837685 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:04.837842 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:04.838224 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:04.838227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:04.838236 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 17:58:04.838397 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:04.838404 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:04.839283 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:04.839288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:04.840405 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:04.840418 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:04.841038 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:04.841042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:04.841141 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:04.841214 (+ 73us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 17:58:04.862667 (+ 21453us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36037/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 17:58:04.862937 (+ 270us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.118.62:35743: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36037/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":137,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.7.118.62:35743: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36037/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 17:58:04.865357 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 12763
2026-09-29T17:58:04Z chronyd exiting
I20260929 17:58:04.902784 7640 test_util.cc:175] -----------------------------------------------
I20260929 17:58:04.902885 7640 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704657566865-7640-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (370 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 17:58:04.903967 7640 mini_oidc.cc:152] Starting JWKS server
I20260929 17:58:04.904811 7640 webserver.cc:460] Webserver started at https://127.0.0.1:36839/ using document root <none> and password file <none>
I20260929 17:58:04.904896 7640 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 17:58:04.905117 7640 webserver.cc:460] Webserver started at http://127.0.0.1:35551/ using document root <none> and password file <none>
2026-09-29T17:58:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:58:04Z Disabled control of system clock
I20260929 17:58:04.915797 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:45537
--webserver_interface=127.7.118.62
--webserver_port=0
--builtin_ntp_servers=127.7.118.20:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:45537
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36839/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704657566865-7640-0/testchainca.cert with env {}
W20260929 17:58:05.046789 12866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:05.047361 12866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:05.047435 12866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:05.051946 12866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:58:05.052259 12866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:05.052284 12866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:05.052424 12866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:58:05.052459 12866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 17:58:05.052495 12866 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 17:58:05.052511 12866 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36839/jwks.json
I20260929 17:58:05.059355 12866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:45537
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:45537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36839/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704657566865-7640-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:05.060616 12866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:05.061842 12866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:05.072178 12872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:05.072577 12874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:05.072757 12871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:05.072868 12866 server_base.cc:1034] running on GCE node
I20260929 17:58:05.073818 12866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:05.075265 12866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:05.076488 12866 hybrid_clock.cc:648] HybridClock initialized: now 1790704685076447 us; error 61 us; skew 500 ppm
I20260929 17:58:05.079479 12866 webserver.cc:460] Webserver started at http://127.7.118.62:36923/ using document root <none> and password file <none>
I20260929 17:58:05.080252 12866 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:05.080335 12866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:05.080580 12866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:05.083269 12866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "387f16fc5b91495f9fe786b4cd3007b7"
format_stamp: "Formatted at 2026-09-29 17:58:05 on dist-test-slave-r7ph"
server_key: "b5bd7e152094645783f3585db99a3040"
server_key_iv: "8e5763fe04aac2d8a578333f66d0e6c8"
server_key_version: "encryptionkey@0"
I20260929 17:58:05.084069 12866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "387f16fc5b91495f9fe786b4cd3007b7"
format_stamp: "Formatted at 2026-09-29 17:58:05 on dist-test-slave-r7ph"
server_key: "b5bd7e152094645783f3585db99a3040"
server_key_iv: "8e5763fe04aac2d8a578333f66d0e6c8"
server_key_version: "encryptionkey@0"
I20260929 17:58:05.089636 12866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 17:58:05.093011 12880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:05.095065 12866 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 17:58:05.095274 12866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "387f16fc5b91495f9fe786b4cd3007b7"
format_stamp: "Formatted at 2026-09-29 17:58:05 on dist-test-slave-r7ph"
server_key: "b5bd7e152094645783f3585db99a3040"
server_key_iv: "8e5763fe04aac2d8a578333f66d0e6c8"
server_key_version: "encryptionkey@0"
I20260929 17:58:05.095453 12866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:05.124855 12866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:05.125686 12866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:05.126130 12866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:05.136049 12866 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:45537
I20260929 17:58:05.136046 12932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:45537 every 8 connection(s)
I20260929 17:58:05.137601 12866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 17:58:05.141479 12933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:05.144479 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 12866
I20260929 17:58:05.144637 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 17:58:05.144979 7640 external_mini_cluster.cc:1428] Setting key 9f97543f0abe4e7da9d9727793b01a6a
I20260929 17:58:05.149549 12933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 387f16fc5b91495f9fe786b4cd3007b7: Bootstrap starting.
I20260929 17:58:05.152736 12933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 387f16fc5b91495f9fe786b4cd3007b7: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:05.153798 12933 log.cc:824] T 00000000000000000000000000000000 P 387f16fc5b91495f9fe786b4cd3007b7: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:05.156791 12933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 387f16fc5b91495f9fe786b4cd3007b7: No bootstrap required, opened a new log
I20260929 17:58:05.157327 12936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:05.146869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50214 (local address 127.7.118.62:45537)
0929 17:58:05.147595 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:05.147613 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:05.147656 (+ 43us) server_negotiation.cc:407] Connection header received
0929 17:58:05.148371 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:05.148393 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:05.148892 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:05.149241 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:05.150045 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:05.151194 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:05.152580 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:05.153306 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:05.153455 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:05.154810 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:05.154816 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:05.154832 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:58:05.155474 (+ 642us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:05.155559 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:58:05.155591 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:05.155719 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:05.155930 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 17:58:05.156270 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":177,"threads_started":1}
I20260929 17:58:05.161568 7640 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:58:05.161586 12933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 387f16fc5b91495f9fe786b4cd3007b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387f16fc5b91495f9fe786b4cd3007b7" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45537 } }
I20260929 17:58:05.161950 12933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 387f16fc5b91495f9fe786b4cd3007b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:05.162050 12933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 387f16fc5b91495f9fe786b4cd3007b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 387f16fc5b91495f9fe786b4cd3007b7, State: Initialized, Role: FOLLOWER
I20260929 17:58:05.162734 12933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 387f16fc5b91495f9fe786b4cd3007b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387f16fc5b91495f9fe786b4cd3007b7" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45537 } }
I20260929 17:58:05.162957 12933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 387f16fc5b91495f9fe786b4cd3007b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:58:05.163038 12933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 387f16fc5b91495f9fe786b4cd3007b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:58:05.163148 12933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 387f16fc5b91495f9fe786b4cd3007b7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:05.164443 12933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 387f16fc5b91495f9fe786b4cd3007b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387f16fc5b91495f9fe786b4cd3007b7" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45537 } }
I20260929 17:58:05.164855 12933 leader_election.cc:302] T 00000000000000000000000000000000 P 387f16fc5b91495f9fe786b4cd3007b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 387f16fc5b91495f9fe786b4cd3007b7; no voters:
I20260929 17:58:05.165266 12933 leader_election.cc:288] T 00000000000000000000000000000000 P 387f16fc5b91495f9fe786b4cd3007b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:58:05.165388 12938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 387f16fc5b91495f9fe786b4cd3007b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:58:05.165647 12938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 387f16fc5b91495f9fe786b4cd3007b7 [term 1 LEADER]: Becoming Leader. State: Replica: 387f16fc5b91495f9fe786b4cd3007b7, State: Running, Role: LEADER
I20260929 17:58:05.166021 12938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 387f16fc5b91495f9fe786b4cd3007b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387f16fc5b91495f9fe786b4cd3007b7" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45537 } }
I20260929 17:58:05.166862 12933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 387f16fc5b91495f9fe786b4cd3007b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:58:05.167943 12940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 387f16fc5b91495f9fe786b4cd3007b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 387f16fc5b91495f9fe786b4cd3007b7. Latest consensus state: current_term: 1 leader_uuid: "387f16fc5b91495f9fe786b4cd3007b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387f16fc5b91495f9fe786b4cd3007b7" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45537 } } }
I20260929 17:58:05.168076 12940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 387f16fc5b91495f9fe786b4cd3007b7 [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:05.167999 12939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 387f16fc5b91495f9fe786b4cd3007b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "387f16fc5b91495f9fe786b4cd3007b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387f16fc5b91495f9fe786b4cd3007b7" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45537 } } }
I20260929 17:58:05.168128 12939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 387f16fc5b91495f9fe786b4cd3007b7 [sys.catalog]: This master's current role is: LEADER
W20260929 17:58:05.172477 12953 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 387f16fc5b91495f9fe786b4cd3007b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:58:05.172602 12953 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:58:05.172752 12947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:58:05.173681 12947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:58:05.179979 12947 catalog_manager.cc:1347] Generated new cluster ID: ec41690077274d27ab09a6f8153f6d3d
I20260929 17:58:05.180101 12947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:58:05.192615 12947 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:58:05.193722 12947 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:58:05.200011 12947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 387f16fc5b91495f9fe786b4cd3007b7: Generated new TSK 0
I20260929 17:58:05.201061 12947 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 17:58:05.233697 12963 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 17:58:05.233829 12963 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 17:58:05.234575 12936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:05.216532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50216 (local address 127.7.118.62:45537)
0929 17:58:05.216678 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:05.216685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:05.216952 (+ 267us) server_negotiation.cc:407] Connection header received
0929 17:58:05.217093 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:05.217097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:05.217164 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:05.217309 (+ 145us) server_negotiation.cc:226] Negotiated authn=JWT
0929 17:58:05.217683 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:05.218577 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:05.219553 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:05.219961 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:05.220068 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:05.220483 (+ 415us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 17:58:05.234047 (+ 13564us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 17:58:05.234408 (+ 361us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50216: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 17:58:05.234648 12936 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50216: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 17:58:05.234831 12962 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:05.216392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45537 (local address 127.0.0.1:50216)
0929 17:58:05.216838 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:05.216850 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:05.216959 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:05.217408 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:05.217411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:05.217419 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 17:58:05.217543 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:05.217549 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:05.218711 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:05.218715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:05.219410 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:05.219420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:05.220131 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:05.220134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:05.220232 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:05.220263 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 17:58:05.234421 (+ 14158us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 17:58:05.234601 (+ 180us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.118.62:45537: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":123,"threads_started":1}
I20260929 17:58:05.235533 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 12866
2026-09-29T17:58:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (371 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 17:58:05.275447 7640 mini_oidc.cc:152] Starting JWKS server
I20260929 17:58:05.276322 7640 webserver.cc:460] Webserver started at https://127.0.0.1:45963/ using document root <none> and password file <none>
I20260929 17:58:05.276429 7640 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 17:58:05.276671 7640 webserver.cc:460] Webserver started at http://127.0.0.1:44503/ using document root <none> and password file <none>
2026-09-29T17:58:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:58:05Z Disabled control of system clock
I20260929 17:58:05.286232 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:36887
--webserver_interface=127.7.118.62
--webserver_port=0
--builtin_ntp_servers=127.7.118.20:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:36887
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45963/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 17:58:05.409693 12969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:05.410100 12969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:05.410145 12969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:05.414542 12969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:58:05.414633 12969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:05.414649 12969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:05.414666 12969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:58:05.414682 12969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 17:58:05.414700 12969 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 17:58:05.414713 12969 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45963/jwks.json
I20260929 17:58:05.419762 12969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:36887
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:36887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45963/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:05.420943 12969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:05.422430 12969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:05.431560 12974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:05.431555 12977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:05.431546 12975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:05.432098 12969 server_base.cc:1034] running on GCE node
I20260929 17:58:05.432802 12969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:05.434072 12969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:05.435325 12969 hybrid_clock.cc:648] HybridClock initialized: now 1790704685435248 us; error 62 us; skew 500 ppm
I20260929 17:58:05.437537 12969 webserver.cc:460] Webserver started at http://127.7.118.62:39263/ using document root <none> and password file <none>
I20260929 17:58:05.438194 12969 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:05.438329 12969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:05.438580 12969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:05.440848 12969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "3225386203b54bada1df880bae23c202"
format_stamp: "Formatted at 2026-09-29 17:58:05 on dist-test-slave-r7ph"
server_key: "83698e1682c4f8c55efef27f10a9c074"
server_key_iv: "0fa84ca7950c5715ae9a931513caff28"
server_key_version: "encryptionkey@0"
I20260929 17:58:05.441499 12969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "3225386203b54bada1df880bae23c202"
format_stamp: "Formatted at 2026-09-29 17:58:05 on dist-test-slave-r7ph"
server_key: "83698e1682c4f8c55efef27f10a9c074"
server_key_iv: "0fa84ca7950c5715ae9a931513caff28"
server_key_version: "encryptionkey@0"
I20260929 17:58:05.445739 12969 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260929 17:58:05.448702 12983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:05.450533 12969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:58:05.450698 12969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "3225386203b54bada1df880bae23c202"
format_stamp: "Formatted at 2026-09-29 17:58:05 on dist-test-slave-r7ph"
server_key: "83698e1682c4f8c55efef27f10a9c074"
server_key_iv: "0fa84ca7950c5715ae9a931513caff28"
server_key_version: "encryptionkey@0"
I20260929 17:58:05.450839 12969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:05.467504 12969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:05.468323 12969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:05.468546 12969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:05.476707 13035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:36887 every 8 connection(s)
I20260929 17:58:05.476711 12969 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:36887
I20260929 17:58:05.477869 12969 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 17:58:05.481611 13036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:05.483824 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 12969
I20260929 17:58:05.483958 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 17:58:05.484239 7640 external_mini_cluster.cc:1428] Setting key a943a43ca8eed2ef74d4d8553a83ea5e
I20260929 17:58:05.489537 13036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3225386203b54bada1df880bae23c202: Bootstrap starting.
I20260929 17:58:05.492074 13036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3225386203b54bada1df880bae23c202: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:05.493063 13036 log.cc:824] T 00000000000000000000000000000000 P 3225386203b54bada1df880bae23c202: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:05.496299 13036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3225386203b54bada1df880bae23c202: No bootstrap required, opened a new log
I20260929 17:58:05.500384 13039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:05.485932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42330 (local address 127.7.118.62:36887)
0929 17:58:05.486762 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:05.486781 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:05.486823 (+ 42us) server_negotiation.cc:407] Connection header received
0929 17:58:05.487626 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:05.487648 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:05.488152 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:05.488591 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:05.489100 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:05.490358 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:05.491380 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:05.491948 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:05.492139 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:05.493149 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:05.493220 (+ 71us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:05.493242 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:58:05.497921 (+ 4679us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:05.498028 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:58:05.498097 (+ 69us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:05.498791 (+ 694us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:05.498884 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 17:58:05.499249 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":139,"threads_started":1}
I20260929 17:58:05.502916 13036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3225386203b54bada1df880bae23c202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3225386203b54bada1df880bae23c202" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 36887 } }
I20260929 17:58:05.503302 13036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3225386203b54bada1df880bae23c202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:05.503376 13036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3225386203b54bada1df880bae23c202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3225386203b54bada1df880bae23c202, State: Initialized, Role: FOLLOWER
I20260929 17:58:05.503916 13036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3225386203b54bada1df880bae23c202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3225386203b54bada1df880bae23c202" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 36887 } }
I20260929 17:58:05.504083 13036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3225386203b54bada1df880bae23c202 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:58:05.504153 13036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3225386203b54bada1df880bae23c202 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:58:05.504261 13036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3225386203b54bada1df880bae23c202 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:05.504359 7640 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:58:05.505597 13036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3225386203b54bada1df880bae23c202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3225386203b54bada1df880bae23c202" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 36887 } }
I20260929 17:58:05.506016 13036 leader_election.cc:302] T 00000000000000000000000000000000 P 3225386203b54bada1df880bae23c202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3225386203b54bada1df880bae23c202; no voters:
I20260929 17:58:05.506438 13036 leader_election.cc:288] T 00000000000000000000000000000000 P 3225386203b54bada1df880bae23c202 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:58:05.506575 13041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3225386203b54bada1df880bae23c202 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:58:05.506819 13041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3225386203b54bada1df880bae23c202 [term 1 LEADER]: Becoming Leader. State: Replica: 3225386203b54bada1df880bae23c202, State: Running, Role: LEADER
I20260929 17:58:05.507150 13041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3225386203b54bada1df880bae23c202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3225386203b54bada1df880bae23c202" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 36887 } }
I20260929 17:58:05.507783 13036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3225386203b54bada1df880bae23c202 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:58:05.508952 13042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3225386203b54bada1df880bae23c202 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3225386203b54bada1df880bae23c202" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3225386203b54bada1df880bae23c202" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 36887 } } }
I20260929 17:58:05.508991 13043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3225386203b54bada1df880bae23c202 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3225386203b54bada1df880bae23c202. Latest consensus state: current_term: 1 leader_uuid: "3225386203b54bada1df880bae23c202" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3225386203b54bada1df880bae23c202" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 36887 } } }
I20260929 17:58:05.509086 13042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3225386203b54bada1df880bae23c202 [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:05.509102 13043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3225386203b54bada1df880bae23c202 [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:05.509533 13050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:58:05.513238 13050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:58:05.519928 13050 catalog_manager.cc:1347] Generated new cluster ID: 6f5351b71f99418294d9b0e9cab5325a
I20260929 17:58:05.520046 13050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:58:05.529284 13050 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:58:05.530498 13050 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:58:05.537715 13050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3225386203b54bada1df880bae23c202: Generated new TSK 0
I20260929 17:58:05.538763 13050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:58:05.564198 13039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:05.558200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42346 (local address 127.7.118.62:36887)
0929 17:58:05.558385 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:05.558391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:05.558612 (+ 221us) server_negotiation.cc:407] Connection header received
0929 17:58:05.558885 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:05.558889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:05.558965 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:05.559075 (+ 110us) server_negotiation.cc:226] Negotiated authn=JWT
0929 17:58:05.559361 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:05.559911 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:05.561261 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:05.561535 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:05.561633 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:05.562108 (+ 475us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 17:58:05.563763 (+ 1655us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45963/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 17:58:05.564029 (+ 266us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42346: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45963/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":80}
W20260929 17:58:05.564283 13039 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42346: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45963/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 17:58:05.566103 13065 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:05.558106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:36887 (local address 127.0.0.1:42346)
0929 17:58:05.558486 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:05.558508 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:05.558652 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:05.559095 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:05.559099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:05.559108 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 17:58:05.559233 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:05.559240 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:05.560028 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:05.560031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:05.561082 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:05.561094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:05.561685 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:05.561689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:05.561783 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:05.561817 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 17:58:05.563967 (+ 2150us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45963/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 17:58:05.565902 (+ 1935us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.118.62:36887: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45963/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":1637,"thread_start_us":170,"threads_started":1}
I20260929 17:58:05.566874 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 12969
E20260929 17:58:05.575166 13066 webserver.cc:533] Webserver: sslize failed:
E20260929 17:58:05.575284 13066 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T17:58:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (331 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T17:58:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:58:05Z Disabled control of system clock
I20260929 17:58:05.617832 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:40387
--webserver_interface=127.7.118.62
--webserver_port=0
--builtin_ntp_servers=127.7.118.20:42085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:40387
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704657566865-7640-0/testchainca.cert with env {}
W20260929 17:58:05.754210 13070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:05.754771 13070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:05.754823 13070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:05.760462 13070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:58:05.760578 13070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:05.760596 13070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:05.760615 13070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:58:05.760633 13070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 17:58:05.760650 13070 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 17:58:05.760664 13070 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 17:58:05.767946 13070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:42085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:40387
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:40387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704657566865-7640-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:05.769135 13070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:05.770186 13070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:05.781637 13078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:05.781714 13076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:05.781637 13075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:05.782166 13070 server_base.cc:1034] running on GCE node
I20260929 17:58:05.783246 13070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:05.784540 13070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:05.785741 13070 hybrid_clock.cc:648] HybridClock initialized: now 1790704685785720 us; error 41 us; skew 500 ppm
I20260929 17:58:05.788360 13070 webserver.cc:460] Webserver started at http://127.7.118.62:44269/ using document root <none> and password file <none>
I20260929 17:58:05.789072 13070 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:05.789145 13070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:05.789381 13070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:05.791742 13070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "a8f2fc5b422846aba1f3eeda2209b96b"
format_stamp: "Formatted at 2026-09-29 17:58:05 on dist-test-slave-r7ph"
server_key: "7452df56da05be738bceab0f43636c9e"
server_key_iv: "81366f20d8cce82e5c9e92652b28b730"
server_key_version: "encryptionkey@0"
I20260929 17:58:05.792429 13070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "a8f2fc5b422846aba1f3eeda2209b96b"
format_stamp: "Formatted at 2026-09-29 17:58:05 on dist-test-slave-r7ph"
server_key: "7452df56da05be738bceab0f43636c9e"
server_key_iv: "81366f20d8cce82e5c9e92652b28b730"
server_key_version: "encryptionkey@0"
I20260929 17:58:05.797768 13070 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 17:58:05.800987 13084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:05.802860 13070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:58:05.803056 13070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "a8f2fc5b422846aba1f3eeda2209b96b"
format_stamp: "Formatted at 2026-09-29 17:58:05 on dist-test-slave-r7ph"
server_key: "7452df56da05be738bceab0f43636c9e"
server_key_iv: "81366f20d8cce82e5c9e92652b28b730"
server_key_version: "encryptionkey@0"
I20260929 17:58:05.803217 13070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:05.818205 13070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:05.819193 13070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:05.819443 13070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:05.828473 13070 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:40387
I20260929 17:58:05.828476 13136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:40387 every 8 connection(s)
I20260929 17:58:05.829726 13070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 17:58:05.833618 13137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:05.836306 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 13070
I20260929 17:58:05.836496 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 17:58:05.836787 7640 external_mini_cluster.cc:1428] Setting key 5e78f57cf02f9459a1e48125694946b4
I20260929 17:58:05.841216 13137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8f2fc5b422846aba1f3eeda2209b96b: Bootstrap starting.
I20260929 17:58:05.844799 13137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8f2fc5b422846aba1f3eeda2209b96b: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:05.845798 13137 log.cc:824] T 00000000000000000000000000000000 P a8f2fc5b422846aba1f3eeda2209b96b: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:05.848778 13137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8f2fc5b422846aba1f3eeda2209b96b: No bootstrap required, opened a new log
I20260929 17:58:05.849977 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:05.838436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37326 (local address 127.7.118.62:40387)
0929 17:58:05.839292 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:05.839314 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:05.839366 (+ 52us) server_negotiation.cc:407] Connection header received
0929 17:58:05.840618 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:05.840645 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:05.841255 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:05.841885 (+ 630us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:05.842721 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:05.844314 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:05.845523 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:05.846342 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:05.846495 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:05.847570 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:05.847578 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:05.847597 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:58:05.848152 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:05.848232 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:58:05.848261 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:05.848373 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:05.848606 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 17:58:05.848962 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":179,"threads_started":1}
I20260929 17:58:05.853916 7640 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:58:05.853967 13137 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8f2fc5b422846aba1f3eeda2209b96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8f2fc5b422846aba1f3eeda2209b96b" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 40387 } }
I20260929 17:58:05.854516 13137 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8f2fc5b422846aba1f3eeda2209b96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:05.854619 13137 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8f2fc5b422846aba1f3eeda2209b96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8f2fc5b422846aba1f3eeda2209b96b, State: Initialized, Role: FOLLOWER
I20260929 17:58:05.855222 13137 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8f2fc5b422846aba1f3eeda2209b96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8f2fc5b422846aba1f3eeda2209b96b" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 40387 } }
I20260929 17:58:05.855425 13137 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8f2fc5b422846aba1f3eeda2209b96b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:58:05.855517 13137 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8f2fc5b422846aba1f3eeda2209b96b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:58:05.855631 13137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8f2fc5b422846aba1f3eeda2209b96b [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:05.856769 13137 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8f2fc5b422846aba1f3eeda2209b96b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8f2fc5b422846aba1f3eeda2209b96b" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 40387 } }
I20260929 17:58:05.857201 13137 leader_election.cc:302] T 00000000000000000000000000000000 P a8f2fc5b422846aba1f3eeda2209b96b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8f2fc5b422846aba1f3eeda2209b96b; no voters:
I20260929 17:58:05.857574 13137 leader_election.cc:288] T 00000000000000000000000000000000 P a8f2fc5b422846aba1f3eeda2209b96b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:58:05.857846 13142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8f2fc5b422846aba1f3eeda2209b96b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:58:05.858124 13142 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8f2fc5b422846aba1f3eeda2209b96b [term 1 LEADER]: Becoming Leader. State: Replica: a8f2fc5b422846aba1f3eeda2209b96b, State: Running, Role: LEADER
I20260929 17:58:05.858553 13142 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8f2fc5b422846aba1f3eeda2209b96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8f2fc5b422846aba1f3eeda2209b96b" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 40387 } }
I20260929 17:58:05.858995 13137 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8f2fc5b422846aba1f3eeda2209b96b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:58:05.860471 13144 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8f2fc5b422846aba1f3eeda2209b96b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8f2fc5b422846aba1f3eeda2209b96b. Latest consensus state: current_term: 1 leader_uuid: "a8f2fc5b422846aba1f3eeda2209b96b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8f2fc5b422846aba1f3eeda2209b96b" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 40387 } } }
I20260929 17:58:05.860471 13143 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8f2fc5b422846aba1f3eeda2209b96b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8f2fc5b422846aba1f3eeda2209b96b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8f2fc5b422846aba1f3eeda2209b96b" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 40387 } } }
I20260929 17:58:05.860656 13144 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8f2fc5b422846aba1f3eeda2209b96b [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:05.861079 13154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:58:05.861537 13143 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8f2fc5b422846aba1f3eeda2209b96b [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:05.865238 13154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:58:05.873031 13154 catalog_manager.cc:1347] Generated new cluster ID: 179784265d334340b0a9daaf1486a964
I20260929 17:58:05.873433 13154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:58:05.881039 13154 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:58:05.882069 13154 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:58:05.887724 13154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8f2fc5b422846aba1f3eeda2209b96b: Generated new TSK 0
I20260929 17:58:05.888538 13154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:58:06.351645 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:05.905541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37336 (local address 127.7.118.62:40387)
0929 17:58:05.905716 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:05.905721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:05.905877 (+ 156us) server_negotiation.cc:407] Connection header received
0929 17:58:05.906132 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:05.906135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:05.906192 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:05.906370 (+ 178us) server_negotiation.cc:226] Negotiated authn=JWT
0929 17:58:05.906629 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:05.907346 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:05.909310 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:05.909569 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:05.909690 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:05.910141 (+ 451us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 17:58:06.351219 (+441078us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 17:58:06.351522 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37336: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":81}
W20260929 17:58:06.351821 13140 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37336: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 17:58:06.352007 13166 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:05.905403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:40387 (local address 127.0.0.1:37336)
0929 17:58:05.905729 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:05.905745 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:05.905948 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:05.906370 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:05.906373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:05.906381 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 17:58:05.906508 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:05.906514 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:05.908007 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:05.908012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:05.909085 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:05.909099 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:05.909745 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:05.909749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:05.909862 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:05.909899 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 17:58:06.351482 (+441583us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 17:58:06.351677 (+ 195us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.118.62:40387: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":121,"threads_started":1}
I20260929 17:58:06.352866 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 13070
2026-09-29T17:58:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (765 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (133 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (134 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:58:06 dist-test-slave-r7ph krb5kdc[13178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:58:06 dist-test-slave-r7ph krb5kdc[13178](info): set up 2 sockets
Sep 29 17:58:06 dist-test-slave-r7ph krb5kdc[13178](info): commencing operation
krb5kdc: starting...
W20260929 17:58:07.710704 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 17:58:07 dist-test-slave-r7ph krb5kdc[13178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:58:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:58:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 17:58:07.845839 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:45053
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:46531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:45053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:07.961381 13191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:07.961774 13191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:07.961896 13191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:07.966359 13191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:58:07.966493 13191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:07.966526 13191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:07.966583 13191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:58:07.966614 13191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:58:07.972445 13191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:46531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:45053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:45053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:07.973723 13191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:07.974901 13191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:07.985409 13197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:07.985731 13191 server_base.cc:1034] running on GCE node
W20260929 17:58:07.986117 13199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:07.986119 13196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:07.986908 13191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:07.988296 13191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:07.989519 13191 hybrid_clock.cc:648] HybridClock initialized: now 1790704687989488 us; error 50 us; skew 500 ppm
Sep 29 17:58:07 dist-test-slave-r7ph krb5kdc[13178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704687, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:07.998426 13191 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 17:58:07.999559 13191 webserver.cc:460] Webserver started at http://127.7.118.62:33517/ using document root <none> and password file <none>
I20260929 17:58:08.000190 13191 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:08.000262 13191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:08.000451 13191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:08.002888 13191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "07dcb063b9854f1b86cfd0d4903938ac"
format_stamp: "Formatted at 2026-09-29 17:58:07 on dist-test-slave-r7ph"
server_key: "8cc6ea1d30ecf429af9d967912e209a2"
server_key_iv: "d4b392e12965ed280995ca30d8d40182"
server_key_version: "encryptionkey@0"
I20260929 17:58:08.003474 13191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "07dcb063b9854f1b86cfd0d4903938ac"
format_stamp: "Formatted at 2026-09-29 17:58:07 on dist-test-slave-r7ph"
server_key: "8cc6ea1d30ecf429af9d967912e209a2"
server_key_iv: "d4b392e12965ed280995ca30d8d40182"
server_key_version: "encryptionkey@0"
I20260929 17:58:08.007759 13191 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 17:58:08.011205 13206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:08.012856 13191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:58:08.013005 13191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "07dcb063b9854f1b86cfd0d4903938ac"
format_stamp: "Formatted at 2026-09-29 17:58:07 on dist-test-slave-r7ph"
server_key: "8cc6ea1d30ecf429af9d967912e209a2"
server_key_iv: "d4b392e12965ed280995ca30d8d40182"
server_key_version: "encryptionkey@0"
I20260929 17:58:08.013198 13191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:08.027559 13191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:08.031697 13191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:08.031965 13191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:08.040515 13191 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:45053
I20260929 17:58:08.040545 13258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:45053 every 8 connection(s)
I20260929 17:58:08.041597 13191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 17:58:08.043596 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 13191
I20260929 17:58:08.043730 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 17:58:08.044016 7640 external_mini_cluster.cc:1428] Setting key a6ecc0371ac6de0385b7bc5338c82388
I20260929 17:58:08.045034 13259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:08.051187 13259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07dcb063b9854f1b86cfd0d4903938ac: Bootstrap starting.
Sep 29 17:58:08 dist-test-slave-r7ph krb5kdc[13178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:58:08.054054 13259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07dcb063b9854f1b86cfd0d4903938ac: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:08.054965 13259 log.cc:824] T 00000000000000000000000000000000 P 07dcb063b9854f1b86cfd0d4903938ac: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:08.057426 13259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07dcb063b9854f1b86cfd0d4903938ac: No bootstrap required, opened a new log
I20260929 17:58:08.060165 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:08.045604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35188 (local address 127.7.118.62:45053)
0929 17:58:08.046128 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:08.046144 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:08.046184 (+ 40us) server_negotiation.cc:407] Connection header received
0929 17:58:08.046968 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:08.046987 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:08.047359 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:08.047736 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:08.048638 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:08.049652 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:08.050585 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:08.051107 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:08.051304 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:08.054553 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:08.054559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:08.054575 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:08.054613 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:08.056676 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:08.057227 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:08.057233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:08.057239 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:08.057367 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:08.057711 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:08.057715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:08.057717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:08.058186 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:08.058486 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:08.058795 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 17:58:08.059131 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":159,"threads_started":1}
I20260929 17:58:08.061975 13259 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07dcb063b9854f1b86cfd0d4903938ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07dcb063b9854f1b86cfd0d4903938ac" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45053 } }
I20260929 17:58:08.062288 13259 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07dcb063b9854f1b86cfd0d4903938ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:08.062362 13259 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07dcb063b9854f1b86cfd0d4903938ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07dcb063b9854f1b86cfd0d4903938ac, State: Initialized, Role: FOLLOWER
I20260929 17:58:08.062894 13259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07dcb063b9854f1b86cfd0d4903938ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07dcb063b9854f1b86cfd0d4903938ac" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45053 } }
I20260929 17:58:08.063083 13259 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07dcb063b9854f1b86cfd0d4903938ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:58:08.063162 13259 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07dcb063b9854f1b86cfd0d4903938ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:58:08.063261 13259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07dcb063b9854f1b86cfd0d4903938ac [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:08.064335 13259 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07dcb063b9854f1b86cfd0d4903938ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07dcb063b9854f1b86cfd0d4903938ac" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45053 } }
I20260929 17:58:08.064724 13259 leader_election.cc:302] T 00000000000000000000000000000000 P 07dcb063b9854f1b86cfd0d4903938ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07dcb063b9854f1b86cfd0d4903938ac; no voters:
I20260929 17:58:08.065090 13259 leader_election.cc:288] T 00000000000000000000000000000000 P 07dcb063b9854f1b86cfd0d4903938ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:58:08.065308 13264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07dcb063b9854f1b86cfd0d4903938ac [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:58:08.065567 13264 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07dcb063b9854f1b86cfd0d4903938ac [term 1 LEADER]: Becoming Leader. State: Replica: 07dcb063b9854f1b86cfd0d4903938ac, State: Running, Role: LEADER
I20260929 17:58:08.065896 13264 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07dcb063b9854f1b86cfd0d4903938ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07dcb063b9854f1b86cfd0d4903938ac" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45053 } }
I20260929 17:58:08.066540 13259 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07dcb063b9854f1b86cfd0d4903938ac [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:58:08.067787 13266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07dcb063b9854f1b86cfd0d4903938ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07dcb063b9854f1b86cfd0d4903938ac. Latest consensus state: current_term: 1 leader_uuid: "07dcb063b9854f1b86cfd0d4903938ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07dcb063b9854f1b86cfd0d4903938ac" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45053 } } }
I20260929 17:58:08.067780 13265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07dcb063b9854f1b86cfd0d4903938ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07dcb063b9854f1b86cfd0d4903938ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07dcb063b9854f1b86cfd0d4903938ac" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45053 } } }
I20260929 17:58:08.067946 13265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07dcb063b9854f1b86cfd0d4903938ac [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:08.067946 13266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07dcb063b9854f1b86cfd0d4903938ac [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:08.068459 13273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:58:08.071624 13273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:58:08.077762 13273 catalog_manager.cc:1347] Generated new cluster ID: 6df5fd8e6a464c13ba66c1c82efa8218
I20260929 17:58:08.077858 13273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:58:08.104502 13273 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:58:08.105538 13273 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:58:08.118410 13273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07dcb063b9854f1b86cfd0d4903938ac: Generated new TSK 0
I20260929 17:58:08.119508 13273 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.118.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
I20260929 17:58:08.164376 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:0
--local_ip_for_outbound_sockets=127.7.118.1
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:45053
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:46531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:08.289788 13285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:08.290140 13285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:08.290190 13285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:08.295711 13285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:08.295821 13285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:08.295895 13285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 17:58:08.302802 13285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:46531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:45053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:08.304373 13285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:08.305547 13285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:08.318212 13293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:08.318154 13291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:08.318426 13290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:08.318591 13285 server_base.cc:1034] running on GCE node
I20260929 17:58:08.319221 13285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:08.320031 13285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:08.321331 13285 hybrid_clock.cc:648] HybridClock initialized: now 1790704688321309 us; error 40 us; skew 500 ppm
Sep 29 17:58:08 dist-test-slave-r7ph krb5kdc[13178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704688, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:08.325100 13285 init.cc:377] Logged in from keytab as kudu/127.7.118.1@KRBTEST.COM (short username kudu)
I20260929 17:58:08.326387 13285 webserver.cc:460] Webserver started at http://127.7.118.1:39983/ using document root <none> and password file <none>
I20260929 17:58:08.327099 13285 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:08.327271 13285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:08.327543 13285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:08.330016 13285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-0/data/instance:
uuid: "ac0bb3c4a201441082439c735a21974f"
format_stamp: "Formatted at 2026-09-29 17:58:08 on dist-test-slave-r7ph"
server_key: "f9eb01a2f324cfc8d9ac0d86705876c0"
server_key_iv: "0d7e363403910942899ccfc80fbe7f3d"
server_key_version: "encryptionkey@0"
I20260929 17:58:08.330821 13285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance:
uuid: "ac0bb3c4a201441082439c735a21974f"
format_stamp: "Formatted at 2026-09-29 17:58:08 on dist-test-slave-r7ph"
server_key: "f9eb01a2f324cfc8d9ac0d86705876c0"
server_key_iv: "0d7e363403910942899ccfc80fbe7f3d"
server_key_version: "encryptionkey@0"
I20260929 17:58:08.335445 13285 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:58:08.338810 13300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:08.340528 13285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:58:08.340734 13285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "ac0bb3c4a201441082439c735a21974f"
format_stamp: "Formatted at 2026-09-29 17:58:08 on dist-test-slave-r7ph"
server_key: "f9eb01a2f324cfc8d9ac0d86705876c0"
server_key_iv: "0d7e363403910942899ccfc80fbe7f3d"
server_key_version: "encryptionkey@0"
I20260929 17:58:08.340880 13285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:08.359290 13285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:08.363312 13285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:08.363567 13285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:08.364388 13285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:08.365401 13285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:08.365479 13285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:08.365553 13285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:08.365593 13285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:08.375823 13285 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:34487
I20260929 17:58:08.375877 13413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:34487 every 8 connection(s)
I20260929 17:58:08.376888 13285 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 17:58:08.383059 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 13285
I20260929 17:58:08.383225 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance
I20260929 17:58:08.383507 7640 external_mini_cluster.cc:1428] Setting key d3c12b88d90ee5e2f38627ac5a725cea
Sep 29 17:58:08 dist-test-slave-r7ph krb5kdc[13178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704688, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.118.2@KRBTEST.COM; defaulting to no policy
I20260929 17:58:08.393990 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:08.379184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:40735 (local address 127.7.118.62:45053)
0929 17:58:08.379395 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:08.379401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:08.380410 (+ 1009us) server_negotiation.cc:407] Connection header received
0929 17:58:08.381992 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:08.381995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:08.382045 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:08.382131 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:08.383786 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:08.384403 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:08.385931 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:08.386220 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:08.386380 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:08.389585 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:08.389590 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:08.389593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:08.389631 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:08.391357 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:08.392043 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:08.392047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:08.392049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:08.392148 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:08.392643 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:08.392646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:08.392647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:08.392861 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:08.392981 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:08.393607 (+ 626us) server_negotiation.cc:299] Negotiation successful
0929 17:58:08.393787 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260929 17:58:08.395107 13416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:08.379646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45053 (local address 127.7.118.1:40735)
0929 17:58:08.380193 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:08.380283 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:08.381716 (+ 1433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:08.382448 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:08.382457 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:08.382884 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:08.383569 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:08.383608 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:08.384561 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:08.384564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:08.385784 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:08.385796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:08.386434 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:08.386438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:08.386636 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:08.387364 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:08.387384 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:08.389359 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:08.391522 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:08.391529 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:08.391551 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:08.391885 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:08.392271 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:08.392275 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:08.392277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:08.392473 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:08.392982 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:08.392991 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:08.393320 (+ 329us) client_negotiation.cc:769] Sending connection context
0929 17:58:08.393548 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 17:58:08.394003 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":134,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.2@KRBTEST.COM" created.
I20260929 17:58:08.396526 13414 heartbeater.cc:347] Connected to a master server at 127.7.118.62:45053
I20260929 17:58:08.396828 13414 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:08.397532 13414 heartbeater.cc:511] Master 127.7.118.62:45053 requested a full tablet report, sending...
I20260929 17:58:08.399453 13223 ts_manager.cc:194] Registered new tserver with Master: ac0bb3c4a201441082439c735a21974f (127.7.118.1:34487)
I20260929 17:58:08.401402 13223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.1@KRBTEST.COM'} at 127.7.118.1:40735
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
I20260929 17:58:08.418380 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:0
--local_ip_for_outbound_sockets=127.7.118.2
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:45053
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:46531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:08.544466 13419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:08.544790 13419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:08.544837 13419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:08.549352 13419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:08.549446 13419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:08.549553 13419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 17:58:08.555404 13419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:46531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:45053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:08.556712 13419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:08.557854 13419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:08.569525 13425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:08.569583 13427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:08.569525 13424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:08.569597 13419 server_base.cc:1034] running on GCE node
I20260929 17:58:08.570526 13419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:08.571307 13419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:08.572566 13419 hybrid_clock.cc:648] HybridClock initialized: now 1790704688572524 us; error 60 us; skew 500 ppm
Sep 29 17:58:08 dist-test-slave-r7ph krb5kdc[13178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704688, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:08.576144 13419 init.cc:377] Logged in from keytab as kudu/127.7.118.2@KRBTEST.COM (short username kudu)
I20260929 17:58:08.577270 13419 webserver.cc:460] Webserver started at http://127.7.118.2:36361/ using document root <none> and password file <none>
I20260929 17:58:08.577900 13419 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:08.577970 13419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:08.578207 13419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:08.580646 13419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-1/data/instance:
uuid: "5d985f381fc7499d978718d72e4ebd47"
format_stamp: "Formatted at 2026-09-29 17:58:08 on dist-test-slave-r7ph"
server_key: "a6f1e6710306a4a33e668bbb0dfa6a81"
server_key_iv: "f083be31fc9a02806291c1045b25c3e7"
server_key_version: "encryptionkey@0"
I20260929 17:58:08.581310 13419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance:
uuid: "5d985f381fc7499d978718d72e4ebd47"
format_stamp: "Formatted at 2026-09-29 17:58:08 on dist-test-slave-r7ph"
server_key: "a6f1e6710306a4a33e668bbb0dfa6a81"
server_key_iv: "f083be31fc9a02806291c1045b25c3e7"
server_key_version: "encryptionkey@0"
I20260929 17:58:08.585722 13419 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 17:58:08.588896 13434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:08.590512 13419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:58:08.590703 13419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "5d985f381fc7499d978718d72e4ebd47"
format_stamp: "Formatted at 2026-09-29 17:58:08 on dist-test-slave-r7ph"
server_key: "a6f1e6710306a4a33e668bbb0dfa6a81"
server_key_iv: "f083be31fc9a02806291c1045b25c3e7"
server_key_version: "encryptionkey@0"
I20260929 17:58:08.590869 13419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:08.612704 13419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:08.616881 13419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:08.617139 13419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:08.617965 13419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:08.619068 13419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:08.619163 13419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:08.619227 13419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:08.619271 13419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:08.629719 13419 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:33035
I20260929 17:58:08.629751 13547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:33035 every 8 connection(s)
I20260929 17:58:08.630865 13419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 17:58:08.636857 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 13419
I20260929 17:58:08.636974 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance
I20260929 17:58:08.637321 7640 external_mini_cluster.cc:1428] Setting key 8cdbcc5b292c8e89144ca19127d040ab
Sep 29 17:58:08 dist-test-slave-r7ph krb5kdc[13178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704688, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.118.3@KRBTEST.COM; defaulting to no policy
I20260929 17:58:08.648674 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:08.633019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:58375 (local address 127.7.118.62:45053)
0929 17:58:08.633210 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:08.633215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:08.634294 (+ 1079us) server_negotiation.cc:407] Connection header received
0929 17:58:08.635838 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:08.635842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:08.635895 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:08.635983 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:08.637404 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:08.638022 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:08.639712 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:08.640127 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:08.640267 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:08.643429 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:08.643435 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:08.643439 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:08.643482 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:08.645345 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:08.646401 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:08.646405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:08.646406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:08.646508 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:08.647077 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:08.647081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:08.647082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:08.647348 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:08.647446 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:08.648383 (+ 937us) server_negotiation.cc:299] Negotiation successful
0929 17:58:08.648505 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 17:58:08.649799 13550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:08.633552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45053 (local address 127.7.118.2:58375)
0929 17:58:08.634110 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:08.634162 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:08.635569 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:08.636172 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:08.636180 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:08.636600 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:08.637228 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:08.637246 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:08.638217 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:08.638221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:08.639577 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:08.639588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:08.640305 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:08.640312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:08.640565 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:08.641385 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:08.641405 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:08.643168 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:08.645510 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:08.645522 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:08.645549 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:08.646089 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:08.646622 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:08.646626 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:08.646629 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:08.646918 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:08.647481 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:08.647492 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:08.647953 (+ 461us) client_negotiation.cc:769] Sending connection context
0929 17:58:08.648255 (+ 302us) client_negotiation.cc:240] Negotiation successful
0929 17:58:08.648712 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":157,"threads_started":1}
I20260929 17:58:08.651311 13548 heartbeater.cc:347] Connected to a master server at 127.7.118.62:45053
I20260929 17:58:08.651616 13548 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.3@KRBTEST.COM" created.
I20260929 17:58:08.652266 13548 heartbeater.cc:511] Master 127.7.118.62:45053 requested a full tablet report, sending...
I20260929 17:58:08.653597 13223 ts_manager.cc:194] Registered new tserver with Master: 5d985f381fc7499d978718d72e4ebd47 (127.7.118.2:33035)
I20260929 17:58:08.654511 13223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.2@KRBTEST.COM'} at 127.7.118.2:58375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
I20260929 17:58:08.671273 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:0
--local_ip_for_outbound_sockets=127.7.118.3
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:45053
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:46531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:08.795109 13553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:08.795449 13553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:08.795500 13553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:08.800441 13553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:08.800527 13553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:08.800624 13553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 17:58:08.806947 13553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:46531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:45053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:08.808282 13553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:08.809454 13553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:08.820891 13558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:08.820891 13561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:08.821179 13553 server_base.cc:1034] running on GCE node
W20260929 17:58:08.821039 13559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:08.821779 13553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:08.822628 13553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:08.823819 13553 hybrid_clock.cc:648] HybridClock initialized: now 1790704688823805 us; error 34 us; skew 500 ppm
Sep 29 17:58:08 dist-test-slave-r7ph krb5kdc[13178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704688, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:08.827504 13553 init.cc:377] Logged in from keytab as kudu/127.7.118.3@KRBTEST.COM (short username kudu)
I20260929 17:58:08.828634 13553 webserver.cc:460] Webserver started at http://127.7.118.3:39699/ using document root <none> and password file <none>
I20260929 17:58:08.829298 13553 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:08.829371 13553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:08.829600 13553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:08.832116 13553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-2/data/instance:
uuid: "5ce7cfc3d1eb4bf3b68f72e5dad66df8"
format_stamp: "Formatted at 2026-09-29 17:58:08 on dist-test-slave-r7ph"
server_key: "426a31acb7019c84809fe8064e5f4c2e"
server_key_iv: "397443f75a1a89afb34c7bd19da33070"
server_key_version: "encryptionkey@0"
I20260929 17:58:08.832803 13553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance:
uuid: "5ce7cfc3d1eb4bf3b68f72e5dad66df8"
format_stamp: "Formatted at 2026-09-29 17:58:08 on dist-test-slave-r7ph"
server_key: "426a31acb7019c84809fe8064e5f4c2e"
server_key_iv: "397443f75a1a89afb34c7bd19da33070"
server_key_version: "encryptionkey@0"
I20260929 17:58:08.837745 13553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 17:58:08.840880 13568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:08.842465 13553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:58:08.842669 13553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "5ce7cfc3d1eb4bf3b68f72e5dad66df8"
format_stamp: "Formatted at 2026-09-29 17:58:08 on dist-test-slave-r7ph"
server_key: "426a31acb7019c84809fe8064e5f4c2e"
server_key_iv: "397443f75a1a89afb34c7bd19da33070"
server_key_version: "encryptionkey@0"
I20260929 17:58:08.842819 13553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:08.869065 13553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:08.872915 13553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:08.873096 13553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:08.873761 13553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:08.874701 13553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:08.874751 13553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:08.874820 13553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:08.874854 13553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:08.884629 13553 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:45023
I20260929 17:58:08.884663 13681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:45023 every 8 connection(s)
I20260929 17:58:08.885622 13553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 17:58:08.889499 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 13553
I20260929 17:58:08.889611 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance
I20260929 17:58:08.889899 7640 external_mini_cluster.cc:1428] Setting key 68401b869d2bb6aeaab5c22c64756604
Sep 29 17:58:08 dist-test-slave-r7ph krb5kdc[13178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704688, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:58:08.901593 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:08.887868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:59733 (local address 127.7.118.62:45053)
0929 17:58:08.888015 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:08.888019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:08.888958 (+ 939us) server_negotiation.cc:407] Connection header received
0929 17:58:08.890686 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:08.890689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:08.890748 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:08.890832 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:08.892102 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:08.892681 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:08.894410 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:08.894681 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:08.894797 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:08.897761 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:08.897764 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:08.897766 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:08.897795 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:08.899358 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:08.899996 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:08.900000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:08.900001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:08.900109 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:08.900479 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:08.900482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:08.900483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:08.900700 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:08.900798 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:08.901332 (+ 534us) server_negotiation.cc:299] Negotiation successful
0929 17:58:08.901466 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 17:58:08.902587 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:08.888290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45053 (local address 127.7.118.3:59733)
0929 17:58:08.888787 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:08.888841 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:08.890454 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:08.891026 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:08.891034 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:08.891433 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:08.891958 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:08.891972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:08.892843 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:08.892850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:08.894218 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:08.894228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:08.894863 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:08.894867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:08.895102 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:08.895832 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:08.895850 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:08.897542 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:08.899502 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:08.899508 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:08.899523 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:08.899858 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:08.900241 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:08.900244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:08.900246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:08.900394 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:08.900809 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:08.900814 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:08.901082 (+ 268us) client_negotiation.cc:769] Sending connection context
0929 17:58:08.901251 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 17:58:08.901545 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":122,"threads_started":1}
I20260929 17:58:08.903793 13682 heartbeater.cc:347] Connected to a master server at 127.7.118.62:45053
I20260929 17:58:08.904057 13682 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:08.904578 13682 heartbeater.cc:511] Master 127.7.118.62:45053 requested a full tablet report, sending...
I20260929 17:58:08.905612 13223 ts_manager.cc:194] Registered new tserver with Master: 5ce7cfc3d1eb4bf3b68f72e5dad66df8 (127.7.118.3:45023)
I20260929 17:58:08.906239 13223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.3@KRBTEST.COM'} at 127.7.118.3:59733
I20260929 17:58:08.916359 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/test-admin.keytab.
Sep 29 17:58:08 dist-test-slave-r7ph krb5kdc[13178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:08.935230 7640 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 17:58:08.938328 13689 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:08.936818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48180 (local address 127.7.118.1:34487)
0929 17:58:08.937127 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:08.937135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:08.937164 (+ 29us) server_negotiation.cc:407] Connection header received
0929 17:58:08.937240 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:08.937246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:08.937441 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:08.937577 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:08.938114 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48180: BlockingRecv error: recv got EOF from 127.0.0.1:48180 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":124,"threads_started":1}
W20260929 17:58:08.938831 7640 init.cc:244] Error while opening credential cache '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/krb5cc)
Sep 29 17:58:08 dist-test-slave-r7ph krb5kdc[13178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:08.940083 7640 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 17:58:08 dist-test-slave-r7ph krb5kdc[13178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:08.941251 7640 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 17:58:08 dist-test-slave-r7ph krb5kdc[13178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.1@KRBTEST.COM
I20260929 17:58:08.951803 13689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:08.942375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48196 (local address 127.7.118.1:34487)
0929 17:58:08.942505 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:08.942513 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:08.942555 (+ 42us) server_negotiation.cc:407] Connection header received
0929 17:58:08.942718 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:08.942721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:08.942793 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:08.942881 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:08.943445 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:08.944215 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:08.945115 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:08.945528 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:08.945650 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:08.948300 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:08.948312 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:08.948326 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:08.948357 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:08.950006 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:08.950557 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:08.950564 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:08.950565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:08.950656 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:08.950887 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:08.950890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:08.950891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:08.951154 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:08.951343 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:08.951507 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 17:58:08.951642 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 17:58:08.956507 13689 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:08.955327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48198 (local address 127.7.118.1:34487)
0929 17:58:08.955462 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:08.955466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:08.955574 (+ 108us) server_negotiation.cc:407] Connection header received
0929 17:58:08.955729 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:08.955737 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:08.955801 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:08.955886 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:08.956387 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48198: BlockingRecv error: recv got EOF from 127.0.0.1:48198 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 17:58:08.956651 7640 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/krb5cc' of type FILE
Sep 29 17:58:08 dist-test-slave-r7ph krb5kdc[13178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:08.957749 7640 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 17:58:08 dist-test-slave-r7ph krb5kdc[13178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:08.958784 7640 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 17:58:08 dist-test-slave-r7ph krb5kdc[13178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.1@KRBTEST.COM
I20260929 17:58:08.968851 13689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:08.959712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48212 (local address 127.7.118.1:34487)
0929 17:58:08.959879 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:08.959883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:08.959946 (+ 63us) server_negotiation.cc:407] Connection header received
0929 17:58:08.960133 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:08.960136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:08.960181 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:08.960263 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:08.960894 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:08.961410 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:08.962400 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:08.962634 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:08.962719 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:08.965401 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:08.965404 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:08.965406 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:08.965437 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:08.967064 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:08.967637 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:08.967640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:08.967641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:08.967723 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:08.967997 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:08.968000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:08.968002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:08.968196 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:08.968354 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:08.968576 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 17:58:08.968690 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260929 17:58:08.972306 13689 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:08.971116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48220 (local address 127.7.118.1:34487)
0929 17:58:08.971265 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:08.971269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:08.971376 (+ 107us) server_negotiation.cc:407] Connection header received
0929 17:58:08.971517 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:08.971520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:08.971575 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:08.971666 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:08.972178 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48220: BlockingRecv error: recv got EOF from 127.0.0.1:48220 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 17:58:08.972379 7640 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704657566865-7640-0/krb5kdc/krb5cc' of type FILE
Sep 29 17:58:08 dist-test-slave-r7ph krb5kdc[13178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:08.973546 7640 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 17:58:08 dist-test-slave-r7ph krb5kdc[13178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:08.974670 7640 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 17:58:08 dist-test-slave-r7ph krb5kdc[13178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.1@KRBTEST.COM
I20260929 17:58:08.985427 13689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:08.975882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48230 (local address 127.7.118.1:34487)
0929 17:58:08.976054 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:08.976060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:08.976190 (+ 130us) server_negotiation.cc:407] Connection header received
0929 17:58:08.976383 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:08.976386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:08.976433 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:08.976526 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:08.977125 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:08.977681 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:08.978654 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:08.978899 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:08.979018 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:08.981768 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:08.981772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:08.981774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:08.981819 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:08.983553 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:08.984101 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:08.984105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:08.984106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:08.984206 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:08.984531 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:08.984534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:08.984536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:08.984731 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:08.984912 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:08.985122 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 17:58:08.985257 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 17:58:08.987629 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 13285
I20260929 17:58:08.994467 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 13419
I20260929 17:58:09.001147 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 13553
I20260929 17:58:09.006994 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 13191
2026-09-29T17:58:09Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2388 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:58:09 dist-test-slave-r7ph krb5kdc[13702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:58:09 dist-test-slave-r7ph krb5kdc[13702](info): set up 2 sockets
Sep 29 17:58:09 dist-test-slave-r7ph krb5kdc[13702](info): commencing operation
krb5kdc: starting...
W20260929 17:58:10.130915 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 17:58:10 dist-test-slave-r7ph krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:58:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:58:10Z Disabled control of system clock
WARNING: no policy specified for oryx/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.62.keytab.
Entry for principal oryx/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.62.keytab.
I20260929 17:58:10.266356 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:45353
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.62.keytab
--principal=oryx/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:45353
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:10.386023 13715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:10.386430 13715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:10.386477 13715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:10.391332 13715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:58:10.391433 13715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:10.391463 13715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:10.391481 13715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:58:10.391497 13715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:58:10.399067 13715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:45353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.62.keytab
--principal=oryx/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:45353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:10.400415 13715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:10.401556 13715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:58:10.411226 13715 server_base.cc:1034] running on GCE node
W20260929 17:58:10.411151 13721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:10.411155 13723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:10.411151 13720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:10.412132 13715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:10.413535 13715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:10.414736 13715 hybrid_clock.cc:648] HybridClock initialized: now 1790704690414719 us; error 38 us; skew 500 ppm
Sep 29 17:58:10 dist-test-slave-r7ph krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704690, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:10.417742 13715 init.cc:377] Logged in from keytab as oryx/127.7.118.62@KRBTEST.COM (short username oryx)
I20260929 17:58:10.418846 13715 webserver.cc:460] Webserver started at http://127.7.118.62:33875/ using document root <none> and password file <none>
I20260929 17:58:10.419487 13715 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:10.419582 13715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:10.419821 13715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:10.422034 13715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "796f18bdc8e94ff69ad6374576106f9f"
format_stamp: "Formatted at 2026-09-29 17:58:10 on dist-test-slave-r7ph"
server_key: "3983a9631adef0824774de5992d3f3c7"
server_key_iv: "340a299fa049711778f075426a726e34"
server_key_version: "encryptionkey@0"
I20260929 17:58:10.422755 13715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "796f18bdc8e94ff69ad6374576106f9f"
format_stamp: "Formatted at 2026-09-29 17:58:10 on dist-test-slave-r7ph"
server_key: "3983a9631adef0824774de5992d3f3c7"
server_key_iv: "340a299fa049711778f075426a726e34"
server_key_version: "encryptionkey@0"
I20260929 17:58:10.426844 13715 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:58:10.429595 13730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:10.431031 13715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:58:10.431231 13715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "796f18bdc8e94ff69ad6374576106f9f"
format_stamp: "Formatted at 2026-09-29 17:58:10 on dist-test-slave-r7ph"
server_key: "3983a9631adef0824774de5992d3f3c7"
server_key_iv: "340a299fa049711778f075426a726e34"
server_key_version: "encryptionkey@0"
I20260929 17:58:10.431375 13715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:10.443542 13715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:10.446686 13715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:10.446898 13715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:10.454346 13715 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:45353
I20260929 17:58:10.454666 13782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:45353 every 8 connection(s)
I20260929 17:58:10.455438 13715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 17:58:10.458761 13783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:10.463516 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 13715
I20260929 17:58:10.463699 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 17:58:10.463981 7640 external_mini_cluster.cc:1428] Setting key 13a9834930f4daa86d5ef473b8f9d9ed
I20260929 17:58:10.464828 13783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f: Bootstrap starting.
I20260929 17:58:10.467527 13783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:10.468320 13783 log.cc:824] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:10.470335 13783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f: No bootstrap required, opened a new log
Sep 29 17:58:10 dist-test-slave-r7ph krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM
I20260929 17:58:10.474452 13783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } }
I20260929 17:58:10.474792 13783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:10.474866 13783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 796f18bdc8e94ff69ad6374576106f9f, State: Initialized, Role: FOLLOWER
I20260929 17:58:10.475445 13783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } }
I20260929 17:58:10.475647 13783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:58:10.475728 13783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:58:10.475849 13783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:10.477053 13783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } }
I20260929 17:58:10.477459 13783 leader_election.cc:302] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 796f18bdc8e94ff69ad6374576106f9f; no voters:
I20260929 17:58:10.477850 13783 leader_election.cc:288] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:58:10.478026 13788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:58:10.478314 13788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [term 1 LEADER]: Becoming Leader. State: Replica: 796f18bdc8e94ff69ad6374576106f9f, State: Running, Role: LEADER
I20260929 17:58:10.478662 13788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } }
I20260929 17:58:10.479306 13783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:58:10.480044 13788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 796f18bdc8e94ff69ad6374576106f9f. Latest consensus state: current_term: 1 leader_uuid: "796f18bdc8e94ff69ad6374576106f9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } }
I20260929 17:58:10.480238 13788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:10.480602 13789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "796f18bdc8e94ff69ad6374576106f9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } }
I20260929 17:58:10.480655 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:10.465527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40672 (local address 127.7.118.62:45353)
0929 17:58:10.466025 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:10.466042 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:10.466080 (+ 38us) server_negotiation.cc:407] Connection header received
0929 17:58:10.466856 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:10.466875 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:10.467363 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:10.467818 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:10.468704 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:10.469706 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:10.470699 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:10.471223 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:10.471408 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:10.474095 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:10.474102 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:10.474188 (+ 86us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:10.474227 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:10.476447 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:10.477083 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:10.477090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:10.477096 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:10.477264 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:10.477586 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:10.477589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:10.477591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:10.478206 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:10.479031 (+ 825us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:10.479200 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 17:58:10.479520 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":175,"threads_started":1}
I20260929 17:58:10.480702 13789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [sys.catalog]: This master's current role is: LEADER
W20260929 17:58:10.484373 13803 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:58:10.484483 13803 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:58:10.484560 13797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:58:10.485317 13797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:58:10.491081 13797 catalog_manager.cc:1347] Generated new cluster ID: dbce91949f94432f864ae3fbb8d5ca2e
I20260929 17:58:10.491186 13797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:58:10.497795 13797 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:58:10.498863 13797 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:58:10.507827 13797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f: Generated new TSK 0
I20260929 17:58:10.508808 13797 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.7.118.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.118.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.118.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.1.keytab.
Entry for principal oryx/127.7.118.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.1.keytab.
I20260929 17:58:10.546142 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:0
--local_ip_for_outbound_sockets=127.7.118.1
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:45353
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.1.keytab
--principal=oryx/127.7.118.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:10.668442 13809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:10.668799 13809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:10.668850 13809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:10.673448 13809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:10.673534 13809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:10.673635 13809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 17:58:10.679587 13809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.1.keytab
--principal=oryx/127.7.118.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:45353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:10.680836 13809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:10.681936 13809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:10.692206 13817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:10.692258 13815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:10.692206 13814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:10.692252 13809 server_base.cc:1034] running on GCE node
I20260929 17:58:10.692956 13809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:10.693634 13809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:10.694847 13809 hybrid_clock.cc:648] HybridClock initialized: now 1790704690694825 us; error 39 us; skew 500 ppm
Sep 29 17:58:10 dist-test-slave-r7ph krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704690, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:10.697860 13809 init.cc:377] Logged in from keytab as oryx/127.7.118.1@KRBTEST.COM (short username oryx)
I20260929 17:58:10.698946 13809 webserver.cc:460] Webserver started at http://127.7.118.1:45531/ using document root <none> and password file <none>
I20260929 17:58:10.699536 13809 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:10.699635 13809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:10.699864 13809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:10.702047 13809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/data/instance:
uuid: "38ba528a27114efbae960672fb79404b"
format_stamp: "Formatted at 2026-09-29 17:58:10 on dist-test-slave-r7ph"
server_key: "a8f741fb900a1b17aa365076ea9a4b6a"
server_key_iv: "b5825b46af57991078324efc81285752"
server_key_version: "encryptionkey@0"
I20260929 17:58:10.702750 13809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance:
uuid: "38ba528a27114efbae960672fb79404b"
format_stamp: "Formatted at 2026-09-29 17:58:10 on dist-test-slave-r7ph"
server_key: "a8f741fb900a1b17aa365076ea9a4b6a"
server_key_iv: "b5825b46af57991078324efc81285752"
server_key_version: "encryptionkey@0"
I20260929 17:58:10.706713 13809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:58:10.709345 13824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:10.710860 13809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:58:10.711018 13809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "38ba528a27114efbae960672fb79404b"
format_stamp: "Formatted at 2026-09-29 17:58:10 on dist-test-slave-r7ph"
server_key: "a8f741fb900a1b17aa365076ea9a4b6a"
server_key_iv: "b5825b46af57991078324efc81285752"
server_key_version: "encryptionkey@0"
I20260929 17:58:10.711164 13809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:10.723308 13809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:10.726689 13809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:10.726904 13809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:10.727522 13809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:10.728438 13809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:10.728507 13809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:10.728576 13809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:10.728631 13809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:10.738827 13809 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:42087
I20260929 17:58:10.738898 13937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:42087 every 8 connection(s)
I20260929 17:58:10.739849 13809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 17:58:10.743551 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 13809
I20260929 17:58:10.743712 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance
I20260929 17:58:10.744017 7640 external_mini_cluster.cc:1428] Setting key 82dd6bd1ba20313d801c7a5cc0b06140
Sep 29 17:58:10 dist-test-slave-r7ph krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704690, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.7.118.2@KRBTEST.COM; defaulting to no policy
I20260929 17:58:10.754886 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:10.741910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:42759 (local address 127.7.118.62:45353)
0929 17:58:10.742042 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:10.742050 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:10.742942 (+ 892us) server_negotiation.cc:407] Connection header received
0929 17:58:10.744410 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:10.744413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:10.744473 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:10.744552 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:10.746061 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:10.746660 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:10.747834 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:10.748071 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:10.748178 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:10.750811 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:10.750815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:10.750817 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:10.750842 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:10.752530 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:10.753181 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:10.753185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:10.753187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:10.753279 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:10.753634 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:10.753637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:10.753639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:10.753831 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:10.753934 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:10.754574 (+ 640us) server_negotiation.cc:299] Negotiation successful
0929 17:58:10.754716 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 17:58:10.755802 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:10.742320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.7.118.1:42759)
0929 17:58:10.742786 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:10.742833 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:10.744160 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:10.744740 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:10.744764 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:10.745373 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:10.745905 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:10.745920 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:10.746795 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:10.746798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:10.747702 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:10.747712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:10.748222 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:10.748226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:10.748402 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:10.749066 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:10.749082 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:10.750604 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:10.752727 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:10.752733 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:10.752752 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:10.753053 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:10.753370 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:10.753373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:10.753375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:10.753534 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:10.753934 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:10.753942 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:10.754324 (+ 382us) client_negotiation.cc:769] Sending connection context
0929 17:58:10.754536 (+ 212us) client_negotiation.cc:240] Negotiation successful
0929 17:58:10.754905 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.118.2@KRBTEST.COM" created.
I20260929 17:58:10.757243 13938 heartbeater.cc:347] Connected to a master server at 127.7.118.62:45353
I20260929 17:58:10.757511 13938 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:10.758067 13938 heartbeater.cc:511] Master 127.7.118.62:45353 requested a full tablet report, sending...
I20260929 17:58:10.759861 13747 ts_manager.cc:194] Registered new tserver with Master: 38ba528a27114efbae960672fb79404b (127.7.118.1:42087)
I20260929 17:58:10.761054 13747 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.118.1@KRBTEST.COM'} at 127.7.118.1:42759
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.118.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.2.keytab.
Entry for principal oryx/127.7.118.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.2.keytab.
I20260929 17:58:10.774606 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:0
--local_ip_for_outbound_sockets=127.7.118.2
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:45353
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.2.keytab
--principal=oryx/127.7.118.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:10.889504 13943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:10.889779 13943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:10.889858 13943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:10.894068 13943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:10.894162 13943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:10.894227 13943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 17:58:10.899858 13943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.2.keytab
--principal=oryx/127.7.118.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:45353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:10.901037 13943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:10.902117 13943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:10.912372 13948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:10.912561 13951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:10.912406 13949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:10.912386 13943 server_base.cc:1034] running on GCE node
I20260929 17:58:10.913072 13943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:10.913817 13943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:10.914987 13943 hybrid_clock.cc:648] HybridClock initialized: now 1790704690914963 us; error 39 us; skew 500 ppm
Sep 29 17:58:10 dist-test-slave-r7ph krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704690, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:10.918232 13943 init.cc:377] Logged in from keytab as oryx/127.7.118.2@KRBTEST.COM (short username oryx)
I20260929 17:58:10.919401 13943 webserver.cc:460] Webserver started at http://127.7.118.2:37979/ using document root <none> and password file <none>
I20260929 17:58:10.920009 13943 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:10.920089 13943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:10.920305 13943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:10.922636 13943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/data/instance:
uuid: "6d13d7d66af745589d65193cd03d8f52"
format_stamp: "Formatted at 2026-09-29 17:58:10 on dist-test-slave-r7ph"
server_key: "fd0de8744075b5579ab4758f3afcf80d"
server_key_iv: "2f1254bdfab6749095d5a59eb500fc84"
server_key_version: "encryptionkey@0"
I20260929 17:58:10.923254 13943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance:
uuid: "6d13d7d66af745589d65193cd03d8f52"
format_stamp: "Formatted at 2026-09-29 17:58:10 on dist-test-slave-r7ph"
server_key: "fd0de8744075b5579ab4758f3afcf80d"
server_key_iv: "2f1254bdfab6749095d5a59eb500fc84"
server_key_version: "encryptionkey@0"
I20260929 17:58:10.927482 13943 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 17:58:10.930555 13958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:10.932138 13943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 17:58:10.932261 13943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "6d13d7d66af745589d65193cd03d8f52"
format_stamp: "Formatted at 2026-09-29 17:58:10 on dist-test-slave-r7ph"
server_key: "fd0de8744075b5579ab4758f3afcf80d"
server_key_iv: "2f1254bdfab6749095d5a59eb500fc84"
server_key_version: "encryptionkey@0"
I20260929 17:58:10.932369 13943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:10.959933 13943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:10.963650 13943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:10.963838 13943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:10.964432 13943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:10.965279 13943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:10.965337 13943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:10.965380 13943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:10.965395 13943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:10.975054 13943 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:36521
I20260929 17:58:10.975068 14071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:36521 every 8 connection(s)
I20260929 17:58:10.976064 13943 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 17:58:10.982043 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 13943
I20260929 17:58:10.982221 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance
I20260929 17:58:10.982534 7640 external_mini_cluster.cc:1428] Setting key d727c25e6a5f9f7db09e5fa510d6d227
Sep 29 17:58:10 dist-test-slave-r7ph krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704690, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.7.118.3@KRBTEST.COM; defaulting to no policy
I20260929 17:58:10.993731 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:10.978136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:52661 (local address 127.7.118.62:45353)
0929 17:58:10.978350 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:10.978356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:10.979387 (+ 1031us) server_negotiation.cc:407] Connection header received
0929 17:58:10.981114 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:10.981118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:10.981184 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:10.981299 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:10.982866 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:10.983539 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:10.985542 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:10.985846 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:10.986004 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:10.988981 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:10.988984 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:10.988987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:10.989016 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:10.990705 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:10.991557 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:10.991561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:10.991563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:10.991657 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:10.992319 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:10.992322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:10.992323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:10.992610 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:10.992702 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:10.993352 (+ 650us) server_negotiation.cc:299] Negotiation successful
0929 17:58:10.993490 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 17:58:10.994596 14074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:10.978604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.7.118.2:52661)
0929 17:58:10.979174 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:10.979229 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:10.980821 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:10.981502 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:10.981511 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:10.981943 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:10.982695 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:10.982715 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:10.983688 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:10.983691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:10.985352 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:10.985370 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:10.986077 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:10.986081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:10.986329 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:10.987047 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:10.987112 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:10.988751 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:10.990885 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:10.990894 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:10.990917 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:10.991354 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:10.991767 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:10.991771 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:10.991773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:10.992027 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:10.992732 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:10.992743 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:10.993053 (+ 310us) client_negotiation.cc:769] Sending connection context
0929 17:58:10.993236 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 17:58:10.993544 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":121,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.118.3@KRBTEST.COM" created.
I20260929 17:58:10.996395 14072 heartbeater.cc:347] Connected to a master server at 127.7.118.62:45353
I20260929 17:58:10.996718 14072 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:10.997373 14072 heartbeater.cc:511] Master 127.7.118.62:45353 requested a full tablet report, sending...
I20260929 17:58:10.998723 13747 ts_manager.cc:194] Registered new tserver with Master: 6d13d7d66af745589d65193cd03d8f52 (127.7.118.2:36521)
I20260929 17:58:10.999379 13747 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.118.2@KRBTEST.COM'} at 127.7.118.2:52661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.118.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.3.keytab.
Entry for principal oryx/127.7.118.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.3.keytab.
I20260929 17:58:11.014544 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:0
--local_ip_for_outbound_sockets=127.7.118.3
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:45353
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.3.keytab
--principal=oryx/127.7.118.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:11.133500 14077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:11.133828 14077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:11.133874 14077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:11.138679 14077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:11.138774 14077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:11.138840 14077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 17:58:11.145224 14077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.3.keytab
--principal=oryx/127.7.118.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:45353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:11.146395 14077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:11.147449 14077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:11.158421 14083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:11.158574 14077 server_base.cc:1034] running on GCE node
W20260929 17:58:11.158421 14082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:11.158473 14085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:11.159308 14077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:11.160024 14077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:11.161216 14077 hybrid_clock.cc:648] HybridClock initialized: now 1790704691161195 us; error 35 us; skew 500 ppm
Sep 29 17:58:11 dist-test-slave-r7ph krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704691, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:11.164486 14077 init.cc:377] Logged in from keytab as oryx/127.7.118.3@KRBTEST.COM (short username oryx)
I20260929 17:58:11.165542 14077 webserver.cc:460] Webserver started at http://127.7.118.3:36181/ using document root <none> and password file <none>
I20260929 17:58:11.166167 14077 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:11.166320 14077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:11.166566 14077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:11.168902 14077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/data/instance:
uuid: "0dd99fb321114c759e2cf85a6fff9ee7"
format_stamp: "Formatted at 2026-09-29 17:58:11 on dist-test-slave-r7ph"
server_key: "5e9dada01b8be6cd3425403435e0319d"
server_key_iv: "4b55c85868317ef6da470a937262b5fb"
server_key_version: "encryptionkey@0"
I20260929 17:58:11.169571 14077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance:
uuid: "0dd99fb321114c759e2cf85a6fff9ee7"
format_stamp: "Formatted at 2026-09-29 17:58:11 on dist-test-slave-r7ph"
server_key: "5e9dada01b8be6cd3425403435e0319d"
server_key_iv: "4b55c85868317ef6da470a937262b5fb"
server_key_version: "encryptionkey@0"
I20260929 17:58:11.174045 14077 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 17:58:11.177330 14092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:11.178839 14077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:58:11.179003 14077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "0dd99fb321114c759e2cf85a6fff9ee7"
format_stamp: "Formatted at 2026-09-29 17:58:11 on dist-test-slave-r7ph"
server_key: "5e9dada01b8be6cd3425403435e0319d"
server_key_iv: "4b55c85868317ef6da470a937262b5fb"
server_key_version: "encryptionkey@0"
I20260929 17:58:11.179141 14077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:11.197743 14077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:11.201534 14077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:11.201771 14077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:11.202473 14077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:11.203372 14077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:11.203450 14077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:11.203538 14077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:11.203583 14077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:11.213358 14077 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:37857
I20260929 17:58:11.213376 14205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:37857 every 8 connection(s)
I20260929 17:58:11.214540 14077 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 17:58:11.222309 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 14077
I20260929 17:58:11.222450 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance
I20260929 17:58:11.222743 7640 external_mini_cluster.cc:1428] Setting key 74b7878a31a1cce71e0f6a1e1fca1bb7
Sep 29 17:58:11 dist-test-slave-r7ph krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704691, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM
I20260929 17:58:11.231038 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.216788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:54047 (local address 127.7.118.62:45353)
0929 17:58:11.216948 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:11.216953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:11.217922 (+ 969us) server_negotiation.cc:407] Connection header received
0929 17:58:11.219478 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:11.219481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:11.219535 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:11.219621 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:11.221081 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.221618 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.223225 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.223495 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.223608 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.226878 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:11.226882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:11.226885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:11.226923 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:11.228560 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.229153 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.229155 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.229157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.229281 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.229739 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.229742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.229743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.229934 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:11.230010 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:11.230780 (+ 770us) server_negotiation.cc:299] Negotiation successful
0929 17:58:11.230910 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 17:58:11.232108 14208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.217240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.7.118.3:54047)
0929 17:58:11.217748 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:11.217799 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:11.219222 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:11.219891 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:11.219900 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:11.220347 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:11.220920 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.220936 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.221746 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.221749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.223060 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.223074 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.223731 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.223736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.224000 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.224823 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:11.224854 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:11.226598 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:11.228701 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.228707 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.228724 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.229043 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.229400 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.229403 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.229405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.229634 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.230055 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:11.230063 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:11.230445 (+ 382us) client_negotiation.cc:769] Sending connection context
0929 17:58:11.230683 (+ 238us) client_negotiation.cc:240] Negotiation successful
0929 17:58:11.231012 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":118,"threads_started":1}
I20260929 17:58:11.233608 14206 heartbeater.cc:347] Connected to a master server at 127.7.118.62:45353
I20260929 17:58:11.233916 14206 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:11.234624 14206 heartbeater.cc:511] Master 127.7.118.62:45353 requested a full tablet report, sending...
I20260929 17:58:11.236016 13747 ts_manager.cc:194] Registered new tserver with Master: 0dd99fb321114c759e2cf85a6fff9ee7 (127.7.118.3:37857)
I20260929 17:58:11.236919 13747 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.118.3@KRBTEST.COM'} at 127.7.118.3:54047
I20260929 17:58:11.237178 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.7.118.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.118.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.118.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.61.keytab.
Entry for principal oryx/127.7.118.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.61.keytab.
I20260929 17:58:11.270467 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.61:38403
--webserver_interface=127.7.118.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.61.keytab
--principal=oryx/127.7.118.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:45353,127.7.118.61:38403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:11.400446 14211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:11.400835 14211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:11.400913 14211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:11.405815 14211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:58:11.405946 14211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:11.405984 14211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:11.406032 14211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:58:11.406069 14211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:58:11.412495 14211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:45353,127.7.118.61:38403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.61.keytab
--principal=oryx/127.7.118.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.61:38403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.118.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
W20260929 17:58:11.412823 14211 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.7.118.62:45353,127.7.118.61:38403'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 17:58:11.413830 14211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:11.415325 14211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:11.424563 14216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:11.424583 14217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:11.424585 14219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:11.424651 14211 server_base.cc:1034] running on GCE node
I20260929 17:58:11.425424 14211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:11.426769 14211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:11.427964 14211 hybrid_clock.cc:648] HybridClock initialized: now 1790704691427934 us; error 45 us; skew 500 ppm
Sep 29 17:58:11 dist-test-slave-r7ph krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704691, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:11.431147 14211 init.cc:377] Logged in from keytab as oryx/127.7.118.61@KRBTEST.COM (short username oryx)
I20260929 17:58:11.432207 14211 webserver.cc:460] Webserver started at http://127.7.118.61:38831/ using document root <none> and password file <none>
I20260929 17:58:11.432854 14211 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:11.432950 14211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:11.433194 14211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:11.436010 14211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-1/data/instance:
uuid: "9df917bc96f642689729eb7f4803dffa"
format_stamp: "Formatted at 2026-09-29 17:58:11 on dist-test-slave-r7ph"
server_key: "4e818000741893b54c7c8f026c159819"
server_key_iv: "ea224a74f3737368682b624f98cea152"
server_key_version: "encryptionkey@0"
I20260929 17:58:11.436748 14211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-1/wal/instance:
uuid: "9df917bc96f642689729eb7f4803dffa"
format_stamp: "Formatted at 2026-09-29 17:58:11 on dist-test-slave-r7ph"
server_key: "4e818000741893b54c7c8f026c159819"
server_key_iv: "ea224a74f3737368682b624f98cea152"
server_key_version: "encryptionkey@0"
I20260929 17:58:11.440968 14211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:58:11.443704 14226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:11.445142 14211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 17:58:11.445305 14211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-1/wal
uuid: "9df917bc96f642689729eb7f4803dffa"
format_stamp: "Formatted at 2026-09-29 17:58:11 on dist-test-slave-r7ph"
server_key: "4e818000741893b54c7c8f026c159819"
server_key_iv: "ea224a74f3737368682b624f98cea152"
server_key_version: "encryptionkey@0"
I20260929 17:58:11.445436 14211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:11.467334 14211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:11.470925 14211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:11.471150 14211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:11.478818 14211 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.61:38403
I20260929 17:58:11.479235 14278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.61:38403 every 8 connection(s)
I20260929 17:58:11.480054 14211 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-1/data/info.pb
I20260929 17:58:11.483330 14279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:11.487489 14279 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:58:11.487937 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 14211
Sep 29 17:58:11 dist-test-slave-r7ph krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704691, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.61@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM
Sep 29 17:58:11 dist-test-slave-r7ph krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.118.61@KRBTEST.COM
I20260929 17:58:11.505054 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.489779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40686 (local address 127.7.118.62:45353)
0929 17:58:11.489935 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:11.489940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:11.490387 (+ 447us) server_negotiation.cc:407] Connection header received
0929 17:58:11.491334 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:11.491337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:11.491419 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:11.491500 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:11.492466 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.492988 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.494605 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.494992 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.495102 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.499904 (+ 4802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:11.499908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:11.499911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:11.499947 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:11.502505 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.503278 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.503306 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.503308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.503409 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.503782 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.503785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.503787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.504061 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:11.504147 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:11.504710 (+ 563us) server_negotiation.cc:299] Negotiation successful
0929 17:58:11.504869 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 17:58:11.506488 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.489638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58418 (local address 127.7.118.61:38403)
0929 17:58:11.490172 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:11.490190 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:11.490232 (+ 42us) server_negotiation.cc:407] Connection header received
0929 17:58:11.491047 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:11.491084 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:11.491298 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:11.491453 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:11.492316 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.493362 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.494640 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.495355 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.495597 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.499886 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:11.499947 (+ 61us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:11.499975 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:11.500021 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:11.502470 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.503094 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.503101 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.503120 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.503286 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.503654 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.503659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.503660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.504219 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:11.505065 (+ 846us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:11.505181 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 17:58:11.505323 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"spinlock_wait_cycles":50048,"thread_start_us":229,"threads_started":1}
I20260929 17:58:11.506488 14282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.489709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.0.0.1:40686)
0929 17:58:11.490207 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:11.490289 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:11.491077 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:11.491574 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:11.491579 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:11.492011 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:11.492316 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.492329 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.493116 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.493120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.494392 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.494408 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.495362 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.495366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.495592 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.496652 (+ 1060us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:11.496676 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:11.499669 (+ 2993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:11.502695 (+ 3026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.502706 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.502720 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.503121 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.503510 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.503513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.503515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.503668 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.504215 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:11.504235 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:11.504398 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 17:58:11.504582 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 17:58:11.505088 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"spinlock_wait_cycles":6400,"thread_start_us":153,"threads_started":1}
I20260929 17:58:11.508201 14279 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 17:58:11 dist-test-slave-r7ph krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704691, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.61@KRBTEST.COM for oryx/127.7.118.61@KRBTEST.COM
I20260929 17:58:11.517273 14282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.508758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:38403 (local address 127.0.0.1:58420)
0929 17:58:11.508874 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:11.508891 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:11.508967 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:11.509445 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:11.509447 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:11.509590 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:11.509719 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.509726 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.510497 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.510499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.511041 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.511049 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.511531 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.511534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.511634 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.512099 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:11.512111 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:11.513440 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:11.515414 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.515417 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.515419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.515705 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.515988 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.515991 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.515992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.516081 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.516525 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:11.516528 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:11.516647 (+ 119us) client_negotiation.cc:769] Sending connection context
0929 17:58:11.516834 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 17:58:11.517059 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":18}
I20260929 17:58:11.517405 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.509158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58420 (local address 127.7.118.61:38403)
0929 17:58:11.509265 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:11.509268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:11.509279 (+ 11us) server_negotiation.cc:407] Connection header received
0929 17:58:11.509310 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:11.509312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:11.509345 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:11.509414 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:11.509817 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.510409 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.511162 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.511427 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.511519 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.513579 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:11.513582 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:11.513584 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:11.513607 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:11.515241 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.515814 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.515817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.515818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.515896 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.516202 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.516205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.516206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.516407 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:11.516497 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:11.516941 (+ 444us) server_negotiation.cc:299] Negotiation successful
0929 17:58:11.517160 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":43}
I20260929 17:58:11.521814 14279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa: Bootstrap starting.
I20260929 17:58:11.524510 14279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:11.525372 14279 log.cc:824] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:11.527769 14279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa: No bootstrap required, opened a new log
I20260929 17:58:11.531291 14279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } }
I20260929 17:58:11.531560 14279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:11.531651 14279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9df917bc96f642689729eb7f4803dffa, State: Initialized, Role: FOLLOWER
I20260929 17:58:11.532302 14279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } }
I20260929 17:58:11.532910 14285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } } }
I20260929 17:58:11.533109 14285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:58:11.533878 14279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [sys.catalog]: configured and running, proceeding with master startup.
W20260929 17:58:11.539297 14296 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:58:11.539412 14296 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:58:11.541047 14211 master_runner.cc:428] Detected that this master 9df917bc96f642689729eb7f4803dffa is joining an existing cluster
I20260929 17:58:11.541119 14211 master_runner.cc:432] Initiating AddMaster RPC to add 127.7.118.61:38403
I20260929 17:58:11.552053 14303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.543637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:38403 (local address 127.0.0.1:58436)
0929 17:58:11.543869 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:11.543882 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:11.543969 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:11.544210 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:11.544212 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:11.544440 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:11.544542 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.544548 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.545571 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.545582 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.546614 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.546625 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.547142 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.547146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.547243 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.547790 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:11.547809 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:11.548164 (+ 355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:11.550307 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.550313 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.550317 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.550549 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.550893 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.550896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.550898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.550973 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.551380 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:11.551384 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:11.551511 (+ 127us) client_negotiation.cc:769] Sending connection context
0929 17:58:11.551639 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 17:58:11.551811 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":71,"spinlock_wait_cycles":896,"thread_start_us":96,"threads_started":1}
I20260929 17:58:11.552090 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.543524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58436 (local address 127.7.118.61:38403)
0929 17:58:11.543631 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:11.543636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:11.543936 (+ 300us) server_negotiation.cc:407] Connection header received
0929 17:58:11.544072 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:11.544075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:11.544114 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:11.544191 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:11.544658 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.545435 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.546731 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.547040 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.547130 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.548310 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:11.548313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:11.548316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:11.548337 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:11.550135 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.550669 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.550673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.550675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.550793 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.551067 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.551073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.551077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.551283 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:11.551364 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:11.551700 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 17:58:11.551903 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":32,"spinlock_wait_cycles":3328}
I20260929 17:58:11.552539 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.543484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40702 (local address 127.7.118.62:45353)
0929 17:58:11.543615 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:11.543620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:11.543886 (+ 266us) server_negotiation.cc:407] Connection header received
0929 17:58:11.544032 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:11.544038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:11.544097 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:11.544186 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:11.544816 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.545587 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.546527 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.546829 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.546935 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.548323 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:11.548327 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:11.548330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:11.548359 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:11.550419 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.550874 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.550877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.550879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.550975 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.551546 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.551550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.551551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.551795 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:11.551898 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:11.552224 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 17:58:11.552356 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 17:58:11.553063 14302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.543426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.0.0.1:40702)
0929 17:58:11.543797 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:11.543813 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:11.543935 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:11.544309 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:11.544312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:11.544545 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:11.544687 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.544695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.545705 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.545708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.546407 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.546417 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.546963 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.546967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.547061 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.547629 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:11.547645 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:11.548156 (+ 511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:11.550549 (+ 2393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.550553 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.550554 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.550774 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.551391 (+ 617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.551392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.551394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.551453 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.551942 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:11.551946 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:11.552032 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 17:58:11.552754 (+ 722us) client_negotiation.cc:240] Negotiation successful
0929 17:58:11.552912 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":24,"thread_start_us":121,"threads_started":1}
Sep 29 17:58:11 dist-test-slave-r7ph krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704690, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.62@KRBTEST.COM for oryx/127.7.118.61@KRBTEST.COM
WARNING: no policy specified for oryx/127.7.118.60@KRBTEST.COM; defaulting to no policy
I20260929 17:58:11.570135 14304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.557901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:38403 (local address 127.0.0.1:58438)
0929 17:58:11.558859 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:11.558919 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:11.559083 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:11.559438 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:11.559444 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:11.559953 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:11.560128 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.560137 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.561248 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.561251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.562344 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.562358 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.563027 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.563030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.563183 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.563885 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:11.563901 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:11.565956 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:11.568203 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.568217 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.568225 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.568600 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.568922 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.568928 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.568932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.569050 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.569495 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:11.569501 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:11.569623 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 17:58:11.569766 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 17:58:11.569945 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":101,"threads_started":1}
I20260929 17:58:11.570544 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.557971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58438 (local address 127.7.118.61:38403)
0929 17:58:11.558099 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:11.558104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:11.559013 (+ 909us) server_negotiation.cc:407] Connection header received
0929 17:58:11.559222 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:11.559225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:11.559274 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:11.559385 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:11.560276 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.561093 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.562483 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.562884 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.563010 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.566151 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:11.566155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:11.566159 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:11.566194 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:11.568048 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.568724 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.568728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.568730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.568817 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.569153 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.569156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.569158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.569386 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:11.569485 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:11.569837 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 17:58:11.569963 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.118.60@KRBTEST.COM" created.
I20260929 17:58:11.572242 13747 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.7.118.61:38403: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9df917bc96f642689729eb7f4803dffa"
member_type: NON_VOTER
last_known_addr {
host: "127.7.118.61"
port: 38403
}
attrs {
promote: true
}
}
dest_uuid: "796f18bdc8e94ff69ad6374576106f9f"
cas_config_opid_index: -1
I20260929 17:58:11.572994 13747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: NON_VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: true } }
I20260929 17:58:11.575266 14309 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9df917bc96f642689729eb7f4803dffa (127.7.118.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: NON_VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: true } } }
I20260929 17:58:11.576014 14253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:11.577485 14253 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 1 FOLLOWER]: Refusing update from remote peer 796f18bdc8e94ff69ad6374576106f9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 17:58:11.577761 14309 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.7.118.61:38403
I20260929 17:58:11.578116 14309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "796f18bdc8e94ff69ad6374576106f9f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: NON_VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: true } } }
I20260929 17:58:11.578398 14309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:11.578558 14211 master.cc:537] Master@127.7.118.61:38403 shutting down...
I20260929 17:58:11.578675 14309 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: NON_VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 17:58:11.582273 13734 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260929 17:58:11.583316 13734 consensus_peers.cc:487] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f -> Peer 9df917bc96f642689729eb7f4803dffa (127.7.118.61:38403): Couldn't send request to peer 9df917bc96f642689729eb7f4803dffa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 17:58:11.583542 14211 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:58:11.583871 14211 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:58:11.583997 14211 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.118.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.60.keytab.
Entry for principal oryx/127.7.118.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.60.keytab.
I20260929 17:58:11.594601 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.60:45305
--webserver_interface=127.7.118.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.60.keytab
--principal=oryx/127.7.118.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:45353,127.7.118.61:38403,127.7.118.60:45305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:58:11.601109 14211 master.cc:552] Master@127.7.118.61:38403 shutdown complete.
I20260929 17:58:11.601210 14211 master_runner.cc:305] Clearing existing system tablet
I20260929 17:58:11.601468 14211 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:11.601547 14211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:11.602921 14211 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 17:58:11.603875 14314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:11.604156 14211 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:11.604326 14211 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 17:58:11.604367 14211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-1/wal
uuid: "9df917bc96f642689729eb7f4803dffa"
format_stamp: "Formatted at 2026-09-29 17:58:11 on dist-test-slave-r7ph"
server_key: "4e818000741893b54c7c8f026c159819"
server_key_iv: "ea224a74f3737368682b624f98cea152"
server_key_version: "encryptionkey@0"
I20260929 17:58:11.606357 14211 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:58:11.608546 14211 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 17:58:11.608716 14211 log.cc:1196] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 17:58:11.609136 14211 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa: Deleting consensus metadata
I20260929 17:58:11.609344 14211 master_runner.cc:315] Copying system tablet from 127.7.118.62:45353
I20260929 17:58:11.610724 14211 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa: tablet copy: Beginning tablet copy session from remote peer at address 127.7.118.62:45353
I20260929 17:58:11.622334 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.611436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40706 (local address 127.7.118.62:45353)
0929 17:58:11.611582 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:11.611587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:11.614846 (+ 3259us) server_negotiation.cc:407] Connection header received
0929 17:58:11.615023 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:11.615027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:11.615082 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:11.615207 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:11.615812 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.616361 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.617264 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.617503 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.617614 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.618847 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:11.618851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:11.618853 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:11.618880 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:11.620592 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.621065 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.621069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.621071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.621159 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.621426 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.621431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.621433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.621636 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:11.621740 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:11.622003 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 17:58:11.622123 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 17:58:11.622370 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.611358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.0.0.1:40706)
0929 17:58:11.614718 (+ 3360us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:11.614733 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:11.614900 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:11.615242 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:11.615246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:11.615460 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:11.615680 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.615690 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.616486 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.616490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.617141 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.617151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.617628 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.617632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.617729 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.618330 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:11.618349 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:11.618702 (+ 353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:11.620725 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.620728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.620730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.620966 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.621248 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.621252 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.621254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.621338 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.621728 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:11.621731 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:11.621839 (+ 108us) client_negotiation.cc:769] Sending connection context
0929 17:58:11.621944 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 17:58:11.622123 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3272,"thread_start_us":116,"threads_started":1}
I20260929 17:58:11.623271 13767 tablet_copy_service.cc:140] P 796f18bdc8e94ff69ad6374576106f9f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9df917bc96f642689729eb7f4803dffa ({username='oryx', principal='oryx/127.7.118.61@KRBTEST.COM'} at 127.0.0.1:40706)
I20260929 17:58:11.623466 13767 tablet_copy_service.cc:161] P 796f18bdc8e94ff69ad6374576106f9f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9df917bc96f642689729eb7f4803dffa at {username='oryx', principal='oryx/127.7.118.61@KRBTEST.COM'} at 127.0.0.1:40706: session id = 9df917bc96f642689729eb7f4803dffa-00000000000000000000000000000000
I20260929 17:58:11.624933 13767 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 17:58:11.626590 14211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 17:58:11.628754 14211 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa: tablet copy: Starting download of 0 data blocks...
I20260929 17:58:11.628965 14211 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa: tablet copy: Starting download of 1 WAL segments...
I20260929 17:58:11.630892 14211 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 17:58:11.632752 13767 tablet_copy_service.cc:342] P 796f18bdc8e94ff69ad6374576106f9f: Request end of tablet copy session 9df917bc96f642689729eb7f4803dffa-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.7.118.61@KRBTEST.COM'} at 127.0.0.1:40706
I20260929 17:58:11.632973 13767 tablet_copy_service.cc:433] P 796f18bdc8e94ff69ad6374576106f9f: ending tablet copy session 9df917bc96f642689729eb7f4803dffa-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9df917bc96f642689729eb7f4803dffa
I20260929 17:58:11.640645 14211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:11.642648 14324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:11.642652 14321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:11.643008 14322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:11.646474 14211 server_base.cc:1034] running on GCE node
I20260929 17:58:11.646715 14211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:11.647089 14211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:11.648243 14211 hybrid_clock.cc:648] HybridClock initialized: now 1790704691648228 us; error 47 us; skew 500 ppm
Sep 29 17:58:11 dist-test-slave-r7ph krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704691, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:11.649883 14211 init.cc:377] Logged in from keytab as oryx/127.7.118.61@KRBTEST.COM (short username oryx)
I20260929 17:58:11.650473 14211 webserver.cc:460] Webserver started at http://127.7.118.61:39833/ using document root <none> and password file <none>
I20260929 17:58:11.650691 14211 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:11.650743 14211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:11.651893 14211 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 17:58:11.652832 14331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:11.653077 14211 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 17:58:11.653151 14211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-1/wal
uuid: "9df917bc96f642689729eb7f4803dffa"
format_stamp: "Formatted at 2026-09-29 17:58:11 on dist-test-slave-r7ph"
server_key: "4e818000741893b54c7c8f026c159819"
server_key_iv: "ea224a74f3737368682b624f98cea152"
server_key_version: "encryptionkey@0"
I20260929 17:58:11.653239 14211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:11.667930 14211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:11.670109 14211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:11.675060 14211 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.61:38403
I20260929 17:58:11.675695 14211 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-1/data/info.pb
I20260929 17:58:11.677459 14384 sys_catalog.cc:263] Verifying existing consensus state
I20260929 17:58:11.678442 14384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa: Bootstrap starting.
I20260929 17:58:11.682425 14383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.61:38403 every 8 connection(s)
I20260929 17:58:11.687508 14384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:58:11.687880 14384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa: Bootstrap complete.
I20260929 17:58:11.688414 14384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: NON_VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: true } }
I20260929 17:58:11.688547 14384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9df917bc96f642689729eb7f4803dffa, State: Initialized, Role: LEARNER
I20260929 17:58:11.688665 14384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: NON_VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: true } }
I20260929 17:58:11.688994 14387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: NON_VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: true } } }
I20260929 17:58:11.689165 14384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:58:11.689181 14387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [sys.catalog]: This master's current role is: LEARNER
I20260929 17:58:11.693368 14398 catalog_manager.cc:1259] Loaded cluster ID: dbce91949f94432f864ae3fbb8d5ca2e
I20260929 17:58:11.693490 14398 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa: loading cluster ID for follower catalog manager: success
I20260929 17:58:11.695281 14398 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa: acquiring CA information for follower catalog manager: success
I20260929 17:58:11.696599 14398 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 17:58:11.761373 14312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:11.761688 14312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:11.761734 14312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 17:58:11.763641 13938 heartbeater.cc:503] Master 127.7.118.62:45353 was elected leader, sending a full tablet report...
W20260929 17:58:11.766844 14312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:58:11.766919 14312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:11.766940 14312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:11.766987 14312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:58:11.767017 14312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:58:11.772766 14312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:45353,127.7.118.61:38403,127.7.118.60:45305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.60.keytab
--principal=oryx/127.7.118.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.60:45305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.118.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:11.774034 14312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:11.775213 14312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:11.784920 14404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:11.784961 14312 server_base.cc:1034] running on GCE node
W20260929 17:58:11.784832 14403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:11.785223 14406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:11.785882 14312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:11.787189 14312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:11.788380 14312 hybrid_clock.cc:648] HybridClock initialized: now 1790704691788356 us; error 50 us; skew 500 ppm
Sep 29 17:58:11 dist-test-slave-r7ph krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704691, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:11.791831 14312 init.cc:377] Logged in from keytab as oryx/127.7.118.60@KRBTEST.COM (short username oryx)
I20260929 17:58:11.792860 14312 webserver.cc:460] Webserver started at http://127.7.118.60:45699/ using document root <none> and password file <none>
I20260929 17:58:11.793439 14312 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:11.793509 14312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:11.793766 14312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:11.796090 14312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-2/data/instance:
uuid: "9d44b2b97944484f88c630984887cf55"
format_stamp: "Formatted at 2026-09-29 17:58:11 on dist-test-slave-r7ph"
server_key: "b4b6ee6bc2974fdb743340d411952304"
server_key_iv: "5486feae2b6456a9b5fb9d021f082b52"
server_key_version: "encryptionkey@0"
I20260929 17:58:11.796674 14312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-2/wal/instance:
uuid: "9d44b2b97944484f88c630984887cf55"
format_stamp: "Formatted at 2026-09-29 17:58:11 on dist-test-slave-r7ph"
server_key: "b4b6ee6bc2974fdb743340d411952304"
server_key_iv: "5486feae2b6456a9b5fb9d021f082b52"
server_key_version: "encryptionkey@0"
I20260929 17:58:11.801015 14312 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:58:11.805437 14413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:11.807237 14312 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 17:58:11.807415 14312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-2/wal
uuid: "9d44b2b97944484f88c630984887cf55"
format_stamp: "Formatted at 2026-09-29 17:58:11 on dist-test-slave-r7ph"
server_key: "b4b6ee6bc2974fdb743340d411952304"
server_key_iv: "5486feae2b6456a9b5fb9d021f082b52"
server_key_version: "encryptionkey@0"
I20260929 17:58:11.807556 14312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:11.840969 14312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:11.845082 14312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:11.845331 14312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:11.853794 14312 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.60:45305
I20260929 17:58:11.854120 14465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.60:45305 every 8 connection(s)
I20260929 17:58:11.855124 14312 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-2/data/info.pb
I20260929 17:58:11.858220 14466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:11.862722 14466 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:58:11.863389 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 14312
Sep 29 17:58:11 dist-test-slave-r7ph krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704691, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.60@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM
Sep 29 17:58:11 dist-test-slave-r7ph krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.118.60@KRBTEST.COM
I20260929 17:58:11.880326 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.865824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40708 (local address 127.7.118.62:45353)
0929 17:58:11.865948 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:11.865952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:11.866090 (+ 138us) server_negotiation.cc:407] Connection header received
0929 17:58:11.866709 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:11.866713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:11.866760 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:11.866860 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:11.868102 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.868984 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.870739 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.871092 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.871219 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.874973 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:11.874980 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:11.874984 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:11.875029 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:11.877370 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.878152 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.878156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.878158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.878290 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.878759 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.878762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.878764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.879055 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:11.879168 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:11.879893 (+ 725us) server_negotiation.cc:299] Negotiation successful
0929 17:58:11.880141 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 17:58:11.881505 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.865596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.0.0.1:40708)
0929 17:58:11.865934 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:11.865991 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:11.866487 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:11.866947 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:11.866953 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:11.867527 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:11.867950 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.867964 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.869184 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.869187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.870605 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.870619 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.871480 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.871486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.871607 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.872489 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:11.872518 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:11.874725 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:11.877512 (+ 2787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.877522 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.877539 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.877968 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.878403 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.878406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.878407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.878619 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.879185 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:11.879191 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:11.879637 (+ 446us) client_negotiation.cc:769] Sending connection context
0929 17:58:11.879820 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 17:58:11.880520 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":131,"spinlock_wait_cycles":3584,"thread_start_us":105,"threads_started":1}
I20260929 17:58:11.881505 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.864934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58536 (local address 127.7.118.60:45305)
0929 17:58:11.865441 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:11.865456 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:11.865493 (+ 37us) server_negotiation.cc:407] Connection header received
0929 17:58:11.866476 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:11.866498 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:11.866761 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:11.866947 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:11.867938 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.869297 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.870633 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.871376 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.871513 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.875547 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:11.875556 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:11.875580 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:11.875612 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:11.877836 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.878496 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.878502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.878504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.878619 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.878993 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.878995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.878996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.879607 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:11.879822 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:11.880103 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 17:58:11.880335 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":366,"spinlock_wait_cycles":2944,"thread_start_us":148,"threads_started":1}
I20260929 17:58:11.883447 14466 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 17:58:11 dist-test-slave-r7ph krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704691, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.60@KRBTEST.COM for oryx/127.7.118.61@KRBTEST.COM
I20260929 17:58:11.893302 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.884162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:38403 (local address 127.0.0.1:58446)
0929 17:58:11.884641 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:11.884656 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:11.884773 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:11.885059 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:11.885062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:11.885281 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:11.885399 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.885405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.886474 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.886485 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.887110 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.887119 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.887632 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.887635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.887756 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.888302 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:11.888320 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:11.889725 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:11.891701 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.891704 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.891706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.891929 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.892223 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.892228 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.892230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.892323 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.892715 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:11.892718 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:11.892825 (+ 107us) client_negotiation.cc:769] Sending connection context
0929 17:58:11.892932 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 17:58:11.893128 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384}
I20260929 17:58:11.893303 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.884233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58446 (local address 127.7.118.61:38403)
0929 17:58:11.884598 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:11.884604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:11.884721 (+ 117us) server_negotiation.cc:407] Connection header received
0929 17:58:11.884883 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:11.884886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:11.884938 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:11.885035 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:11.885526 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.886349 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.887253 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.887504 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.887635 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.889897 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:11.889900 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:11.889902 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:11.889930 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:11.891566 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.892036 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.892040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.892046 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.892133 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.892412 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.892416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.892418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.892622 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:11.892727 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:11.892986 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 17:58:11.893140 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":112,"threads_started":1}
I20260929 17:58:11.894599 14466 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.60" port: 45305 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 17:58:11 dist-test-slave-r7ph krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704691, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.60@KRBTEST.COM for oryx/127.7.118.60@KRBTEST.COM
I20260929 17:58:11.904731 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.895502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58552 (local address 127.7.118.60:45305)
0929 17:58:11.895606 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:11.895609 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:11.895637 (+ 28us) server_negotiation.cc:407] Connection header received
0929 17:58:11.895668 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:11.895670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:11.895707 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:11.895771 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:11.896436 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.897400 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.898383 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.898865 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.898973 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.901403 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:11.901406 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:11.901408 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:11.901428 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:11.902979 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.903444 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.903450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.903452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.903525 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.903817 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.903820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.903822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.904016 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:11.904123 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:11.904363 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 17:58:11.904568 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":38}
I20260929 17:58:11.904731 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.895329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:45305 (local address 127.0.0.1:58552)
0929 17:58:11.895493 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:11.895511 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:11.895585 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:11.896004 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:11.896006 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:11.896161 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:11.896246 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.896251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.897568 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.897571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.898184 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.898193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.899039 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.899052 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.899131 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.899744 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:11.899757 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:11.901241 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:11.903120 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.903123 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.903125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.903349 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.903642 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.903646 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.903648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.903730 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.904127 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:11.904130 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:11.904219 (+ 89us) client_negotiation.cc:769] Sending connection context
0929 17:58:11.904321 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 17:58:11.904522 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":5}
I20260929 17:58:11.908038 14466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55: Bootstrap starting.
I20260929 17:58:11.911005 14466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:11.912223 14466 log.cc:824] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:11.914788 14466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55: No bootstrap required, opened a new log
I20260929 17:58:11.918094 14466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 45305 } }
I20260929 17:58:11.918416 14466 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:11.918473 14466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d44b2b97944484f88c630984887cf55, State: Initialized, Role: FOLLOWER
I20260929 17:58:11.918967 14466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 45305 } }
I20260929 17:58:11.919509 14472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 45305 } } }
I20260929 17:58:11.919710 14472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:58:11.920396 14466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 17:58:11.926978 14483 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:58:11.927125 14483 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:58:11.931013 14312 master_runner.cc:428] Detected that this master 9d44b2b97944484f88c630984887cf55 is joining an existing cluster
I20260929 17:58:11.931118 14312 master_runner.cc:432] Initiating AddMaster RPC to add 127.7.118.60:45305
I20260929 17:58:11.936170 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 13809
I20260929 17:58:11.943361 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:42087
--local_ip_for_outbound_sockets=127.7.118.1
--tserver_master_addrs=127.7.118.62:45353
--webserver_port=45531
--webserver_interface=127.7.118.1
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.1.keytab
--principal=oryx/127.7.118.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.118.62:45353,127.7.118.61:38403
--tserver_master_addrs=127.7.118.62:45353,127.7.118.61:38403,127.7.118.60:45305 with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:58:11.960701 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.934006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58448 (local address 127.7.118.61:38403)
0929 17:58:11.934327 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:11.934331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:11.934478 (+ 147us) server_negotiation.cc:407] Connection header received
0929 17:58:11.934785 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:11.934788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:11.934845 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:11.934937 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:11.935792 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.936595 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.938098 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.938516 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.938640 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.940205 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:11.940209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:11.940212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:11.940241 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:11.942294 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.959126 (+ 16832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.959131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.959133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.959264 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.959593 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.959596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.959598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.959848 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:11.959927 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:11.960340 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 17:58:11.960505 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260929 17:58:11.961349 14489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.933860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:38403 (local address 127.0.0.1:58448)
0929 17:58:11.934373 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:11.934389 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:11.934521 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:11.935038 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:11.935044 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:11.935463 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:11.935630 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.935639 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.936897 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.936904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.937959 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.937969 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.938797 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.938800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.938900 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.939515 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:11.939530 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:11.940023 (+ 493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:11.942413 (+ 2390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.942419 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.942424 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.958907 (+ 16483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.959391 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.959394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.959396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.959487 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.960019 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:11.960021 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:11.960141 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 17:58:11.960985 (+ 844us) client_negotiation.cc:240] Negotiation successful
0929 17:58:11.961154 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":1,"thread_start_us":120,"threads_started":1}
I20260929 17:58:11.963572 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.935553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:45305 (local address 127.0.0.1:58564)
0929 17:58:11.936264 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:11.937528 (+ 1264us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:11.937661 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:11.937936 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:11.937938 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:11.938221 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:11.938363 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.938371 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.939350 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.939356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.940220 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.940228 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.940862 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.940865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.940951 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.941961 (+ 1010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:11.941974 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:11.959058 (+ 17084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:11.961693 (+ 2635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.961696 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.961698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.962137 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.962497 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.962500 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.962502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.962577 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.963011 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:11.963014 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:11.963090 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 17:58:11.963208 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 17:58:11.963364 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":173,"threads_started":1}
I20260929 17:58:11.964830 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.934882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58564 (local address 127.7.118.60:45305)
0929 17:58:11.935157 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:11.935162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:11.937599 (+ 2437us) server_negotiation.cc:407] Connection header received
0929 17:58:11.937746 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:11.937749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:11.937782 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:11.937823 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:11.938477 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.939224 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.940337 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.940660 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.940734 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.959264 (+ 18530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:11.959268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:11.959271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:11.959298 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:11.961118 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.962310 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.962313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.962315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.962396 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.962671 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.962674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.962676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.962889 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:11.962975 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:11.964510 (+ 1535us) server_negotiation.cc:299] Negotiation successful
0929 17:58:11.964683 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208}
I20260929 17:58:11.967197 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.933863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.0.0.1:40720)
0929 17:58:11.934865 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:11.934879 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:11.938102 (+ 3223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:11.938625 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:11.938628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:11.938816 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:11.939215 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.939223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.940362 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.940365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.941210 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.941218 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.941818 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.941822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.941950 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.942566 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:11.942586 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:11.960423 (+ 17837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:11.965350 (+ 4927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.965353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.965355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.965626 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.965969 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.965972 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.965974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.966056 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.966573 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:11.966575 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:11.966705 (+ 130us) client_negotiation.cc:769] Sending connection context
0929 17:58:11.966812 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 17:58:11.966976 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":935,"mutex_wait_us":26,"spinlock_wait_cycles":768,"thread_start_us":95,"threads_started":1}
I20260929 17:58:11.967233 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.934015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40720 (local address 127.7.118.62:45353)
0929 17:58:11.934306 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:11.934312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:11.934962 (+ 650us) server_negotiation.cc:407] Connection header received
0929 17:58:11.938274 (+ 3312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:11.938278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:11.938348 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:11.938432 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:11.939387 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.940209 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.941330 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.941683 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.941807 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.963225 (+ 21418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:11.963230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:11.963233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:11.963268 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:11.965190 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.965760 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.965765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.965766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.965861 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.966149 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.966153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.966155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.966425 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:11.966501 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:11.966912 (+ 411us) server_negotiation.cc:299] Negotiation successful
0929 17:58:11.967066 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260929 17:58:11.976905 14472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:58:11.977121 14472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 45305 } }
I20260929 17:58:11.978437 14472 leader_election.cc:288] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 796f18bdc8e94ff69ad6374576106f9f (127.7.118.62:45353), 9df917bc96f642689729eb7f4803dffa (127.7.118.61:38403)
Sep 29 17:58:11 dist-test-slave-r7ph krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704690, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.62@KRBTEST.COM for oryx/127.7.118.60@KRBTEST.COM
I20260929 17:58:11.979266 14358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d44b2b97944484f88c630984887cf55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9df917bc96f642689729eb7f4803dffa" is_pre_election: true
I20260929 17:58:11.979463 14358 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 1 LEARNER]: Handling vote request from an unknown peer 9d44b2b97944484f88c630984887cf55
I20260929 17:58:11.979498 13757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d44b2b97944484f88c630984887cf55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "796f18bdc8e94ff69ad6374576106f9f" is_pre_election: true
I20260929 17:58:11.979609 14358 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 9d44b2b97944484f88c630984887cf55 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260929 17:58:11.980331 14416 leader_election.cc:302] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d44b2b97944484f88c630984887cf55; no voters: 796f18bdc8e94ff69ad6374576106f9f, 9df917bc96f642689729eb7f4803dffa
I20260929 17:58:11.980861 14472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:11.982237 14472 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 17:58:11.987298 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.972031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58580 (local address 127.7.118.60:45305)
0929 17:58:11.972220 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:11.972225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:11.972246 (+ 21us) server_negotiation.cc:407] Connection header received
0929 17:58:11.972410 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:11.972413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:11.972473 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:11.972623 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:11.973416 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.974689 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.976213 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.976564 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.976665 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.982854 (+ 6189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:11.982858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:11.982861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:11.982894 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:11.984894 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.985508 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.985511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.985514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.985687 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:11.986166 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:11.986169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:11.986171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:11.986469 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:11.986570 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:11.986970 (+ 400us) server_negotiation.cc:299] Negotiation successful
0929 17:58:11.987119 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 17:58:11.987299 14304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.971939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:45305 (local address 127.0.0.1:58580)
0929 17:58:11.972157 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:11.972172 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:11.972260 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:11.972700 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:11.972703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:11.972958 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:11.973077 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.973084 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.974874 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.974880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.976004 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.976020 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.976833 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.976840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:11.976956 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:11.977666 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:11.977683 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:11.982662 (+ 4979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:11.985062 (+ 2400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.985067 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.985069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.985381 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.985913 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:11.985917 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:11.985920 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:11.986049 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:11.986591 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:11.986594 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:11.986769 (+ 175us) client_negotiation.cc:769] Sending connection context
0929 17:58:11.986897 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 17:58:11.987107 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260929 17:58:11.989025 13747 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.7.118.60:45305: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9d44b2b97944484f88c630984887cf55"
member_type: NON_VOTER
last_known_addr {
host: "127.7.118.60"
port: 45305
}
attrs {
promote: true
}
}
dest_uuid: "796f18bdc8e94ff69ad6374576106f9f"
cas_config_opid_index: 5
I20260929 17:58:11.989741 13747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: NON_VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: true } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: NON_VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: true } }
I20260929 17:58:11.990988 14308 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9d44b2b97944484f88c630984887cf55 (127.7.118.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: NON_VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: true } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: NON_VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: true } } }
I20260929 17:58:11.992702 14308 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.7.118.60:45305
I20260929 17:58:11.992936 14308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "796f18bdc8e94ff69ad6374576106f9f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: NON_VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: true } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: NON_VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: true } } }
I20260929 17:58:11.993046 14308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:11.994596 14440 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 1 FOLLOWER]: Refusing update from remote peer 796f18bdc8e94ff69ad6374576106f9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 17:58:11.995471 14312 master.cc:537] Master@127.7.118.60:45305 shutting down...
I20260929 17:58:11.995388 14309 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: NON_VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 17:58:11.997902 13732 consensus_peers.cc:487] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f -> Peer 9d44b2b97944484f88c630984887cf55 (127.7.118.60:45305): Couldn't send request to peer 9d44b2b97944484f88c630984887cf55. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 17:58:12.001343 14312 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:58:12.001950 14072 heartbeater.cc:503] Master 127.7.118.62:45353 was elected leader, sending a full tablet report...
I20260929 17:58:12.002456 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.993662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58456 (local address 127.7.118.61:38403)
0929 17:58:11.995071 (+ 1409us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:11.995076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:11.995094 (+ 18us) server_negotiation.cc:407] Connection header received
0929 17:58:11.995148 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:11.995151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:11.995215 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:11.995324 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:11.997060 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.998128 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.000327 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.001057 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.001314 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.001497 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:12.002166 (+ 669us) server_negotiation.cc:299] Negotiation successful
0929 17:58:12.002313 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1311}
I20260929 17:58:12.003033 14304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:11.993555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:38403 (local address 127.0.0.1:58456)
0929 17:58:11.994129 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:11.994153 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:11.994291 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:11.996690 (+ 2399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:11.996696 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:11.996743 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:11.996887 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:11.996897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:11.998380 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:11.998383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.000131 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.000148 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.001713 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.001718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.001833 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.001901 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 17:58:12.002804 (+ 903us) client_negotiation.cc:240] Negotiation successful
0929 17:58:12.002904 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458}
I20260929 17:58:12.003968 14358 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 1 LEARNER]: Refusing update from remote peer 796f18bdc8e94ff69ad6374576106f9f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 17:58:12.001716 14312 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:58:12.004544 14309 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: NON_VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 17:58:12.004668 14312 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55: stopping tablet replica
I20260929 17:58:12.006780 14358 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9d44b2b97944484f88c630984887cf55 (127.7.118.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: NON_VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: true } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: NON_VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: true } } }
I20260929 17:58:12.009040 14494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "796f18bdc8e94ff69ad6374576106f9f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: NON_VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: true } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: NON_VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: true } } }
I20260929 17:58:12.009302 14494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [sys.catalog]: This master's current role is: LEARNER
I20260929 17:58:12.009495 14387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 796f18bdc8e94ff69ad6374576106f9f. Latest consensus state: current_term: 1 leader_uuid: "796f18bdc8e94ff69ad6374576106f9f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: NON_VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: true } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: NON_VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: true } } }
I20260929 17:58:12.009630 14387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [sys.catalog]: This master's current role is: LEARNER
I20260929 17:58:12.010620 14309 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f: attempting to promote NON_VOTER 9df917bc96f642689729eb7f4803dffa to VOTER
I20260929 17:58:12.011142 14309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: NON_VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: true } }
W20260929 17:58:12.013777 13732 consensus_peers.cc:487] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f -> Peer 9d44b2b97944484f88c630984887cf55 (127.7.118.60:45305): Couldn't send request to peer 9d44b2b97944484f88c630984887cf55. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 17:58:12.014139 14358 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 1 LEARNER]: Refusing update from remote peer 796f18bdc8e94ff69ad6374576106f9f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 17:58:12.014631 14308 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 17:58:12.017652 14308 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9df917bc96f642689729eb7f4803dffa (127.7.118.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: NON_VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: true } } }
I20260929 17:58:12.018183 14358 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9df917bc96f642689729eb7f4803dffa (127.7.118.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: NON_VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: true } } }
I20260929 17:58:12.019409 14308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "796f18bdc8e94ff69ad6374576106f9f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: NON_VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: true } } }
I20260929 17:58:12.019529 14308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:12.019641 14308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "796f18bdc8e94ff69ad6374576106f9f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: NON_VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: true } } }
I20260929 17:58:12.019704 14308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:12.020244 14387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "796f18bdc8e94ff69ad6374576106f9f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: NON_VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: true } } }
I20260929 17:58:12.020375 14387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:58:12.021821 14312 master.cc:552] Master@127.7.118.60:45305 shutdown complete.
I20260929 17:58:12.021879 14312 master_runner.cc:305] Clearing existing system tablet
I20260929 17:58:12.022120 14312 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:12.022177 14312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:12.023617 14312 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 17:58:12.024899 14497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:12.025202 14312 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:12.025398 14312 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 17:58:12.025467 14312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-2/wal
uuid: "9d44b2b97944484f88c630984887cf55"
format_stamp: "Formatted at 2026-09-29 17:58:11 on dist-test-slave-r7ph"
server_key: "b4b6ee6bc2974fdb743340d411952304"
server_key_iv: "5486feae2b6456a9b5fb9d021f082b52"
server_key_version: "encryptionkey@0"
I20260929 17:58:12.027921 14312 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:58:12.031381 14312 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 17:58:12.031599 14312 log.cc:1196] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 17:58:12.031914 14312 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55: Deleting consensus metadata
I20260929 17:58:12.032753 14312 master_runner.cc:315] Copying system tablet from 127.7.118.62:45353
I20260929 17:58:12.034332 14312 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55: tablet copy: Beginning tablet copy session from remote peer at address 127.7.118.62:45353
I20260929 17:58:12.047966 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.035082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40726 (local address 127.7.118.62:45353)
0929 17:58:12.035216 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:12.035221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:12.038716 (+ 3495us) server_negotiation.cc:407] Connection header received
0929 17:58:12.038853 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:12.038856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:12.038925 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:12.039030 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:12.039934 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.040754 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.042035 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.042413 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.042557 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.044006 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:12.044010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:12.044013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:12.044045 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:12.045935 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:12.046548 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:12.046552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:12.046554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:12.046658 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:12.046995 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:12.046999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:12.047000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:12.047263 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:12.047366 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:12.047659 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 17:58:12.047800 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 17:58:12.047973 14503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.035028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.0.0.1:40726)
0929 17:58:12.038566 (+ 3538us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:12.038587 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:12.038750 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:12.039110 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:12.039113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:12.039370 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:12.039761 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.039772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.040906 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.040909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.041853 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.041866 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.042567 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.042572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.042682 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.043326 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:12.043344 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:12.043813 (+ 469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:12.046094 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:12.046098 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:12.046100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:12.046437 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:12.046788 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:12.046791 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:12.046794 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:12.046896 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:12.047364 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:12.047367 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:12.047478 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 17:58:12.047605 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 17:58:12.047773 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3419,"thread_start_us":104,"threads_started":1}
I20260929 17:58:12.048908 13767 tablet_copy_service.cc:140] P 796f18bdc8e94ff69ad6374576106f9f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9d44b2b97944484f88c630984887cf55 ({username='oryx', principal='oryx/127.7.118.60@KRBTEST.COM'} at 127.0.0.1:40726)
I20260929 17:58:12.049016 13767 tablet_copy_service.cc:161] P 796f18bdc8e94ff69ad6374576106f9f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9d44b2b97944484f88c630984887cf55 at {username='oryx', principal='oryx/127.7.118.60@KRBTEST.COM'} at 127.0.0.1:40726: session id = 9d44b2b97944484f88c630984887cf55-00000000000000000000000000000000
I20260929 17:58:12.050221 13767 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 17:58:12.051682 14312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 17:58:12.054514 14312 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55: tablet copy: Starting download of 0 data blocks...
I20260929 17:58:12.054744 14312 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55: tablet copy: Starting download of 1 WAL segments...
I20260929 17:58:12.056869 14312 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 17:58:12.059303 13767 tablet_copy_service.cc:342] P 796f18bdc8e94ff69ad6374576106f9f: Request end of tablet copy session 9d44b2b97944484f88c630984887cf55-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.7.118.60@KRBTEST.COM'} at 127.0.0.1:40726
I20260929 17:58:12.059437 13767 tablet_copy_service.cc:433] P 796f18bdc8e94ff69ad6374576106f9f: ending tablet copy session 9d44b2b97944484f88c630984887cf55-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9d44b2b97944484f88c630984887cf55
I20260929 17:58:12.069000 14312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:12.071516 14504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:12.072281 14507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:12.072569 14505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:12.075774 14312 server_base.cc:1034] running on GCE node
I20260929 17:58:12.076207 14312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:12.076598 14312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:12.077782 14312 hybrid_clock.cc:648] HybridClock initialized: now 1790704692077759 us; error 48 us; skew 500 ppm
Sep 29 17:58:12 dist-test-slave-r7ph krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704692, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:12.079962 14312 init.cc:377] Logged in from keytab as oryx/127.7.118.60@KRBTEST.COM (short username oryx)
I20260929 17:58:12.080585 14312 webserver.cc:460] Webserver started at http://127.7.118.60:39247/ using document root <none> and password file <none>
I20260929 17:58:12.080794 14312 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:12.080842 14312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:12.082513 14312 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 17:58:12.083879 14514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:12.084230 14312 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 17:58:12.084359 14312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-2/wal
uuid: "9d44b2b97944484f88c630984887cf55"
format_stamp: "Formatted at 2026-09-29 17:58:11 on dist-test-slave-r7ph"
server_key: "b4b6ee6bc2974fdb743340d411952304"
server_key_iv: "5486feae2b6456a9b5fb9d021f082b52"
server_key_version: "encryptionkey@0"
I20260929 17:58:12.084517 14312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:12.094440 14312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:12.097258 14312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:12.102705 14312 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.60:45305
I20260929 17:58:12.102752 14566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.60:45305 every 8 connection(s)
I20260929 17:58:12.103632 14312 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/master-2/data/info.pb
I20260929 17:58:12.106021 14567 sys_catalog.cc:263] Verifying existing consensus state
I20260929 17:58:12.107263 14567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55: Bootstrap starting.
W20260929 17:58:12.118809 14492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:12.119141 14492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:12.119217 14492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 17:58:12.120911 14567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:58:12.121279 14567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55: Bootstrap complete.
I20260929 17:58:12.122080 14567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: NON_VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: true } }
I20260929 17:58:12.122242 14567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9d44b2b97944484f88c630984887cf55, State: Initialized, Role: LEARNER
I20260929 17:58:12.122387 14567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: NON_VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: true } }
I20260929 17:58:12.122793 14573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: NON_VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: true } } }
I20260929 17:58:12.122921 14567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:58:12.123008 14573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [sys.catalog]: This master's current role is: LEARNER
W20260929 17:58:12.124578 14492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:12.124666 14492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:12.124774 14492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 17:58:12.127497 14584 catalog_manager.cc:1259] Loaded cluster ID: dbce91949f94432f864ae3fbb8d5ca2e
I20260929 17:58:12.127584 14584 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55: loading cluster ID for follower catalog manager: success
I20260929 17:58:12.130149 14584 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55: acquiring CA information for follower catalog manager: success
I20260929 17:58:12.132036 14584 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 17:58:12.132289 14492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.1.keytab
--principal=oryx/127.7.118.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:42087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=45531
--tserver_master_addrs=127.7.118.62:45353,127.7.118.61:38403,127.7.118.60:45305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:12.133419 14492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:12.134560 14492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:12.146914 14589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:12.147066 14586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:12.147112 14492 server_base.cc:1034] running on GCE node
W20260929 17:58:12.147886 14587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:12.148705 14492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:12.149609 14492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:12.150870 14492 hybrid_clock.cc:648] HybridClock initialized: now 1790704692150817 us; error 73 us; skew 500 ppm
Sep 29 17:58:12 dist-test-slave-r7ph krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704692, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:12.155371 14492 init.cc:377] Logged in from keytab as oryx/127.7.118.1@KRBTEST.COM (short username oryx)
I20260929 17:58:12.157037 14492 webserver.cc:460] Webserver started at http://127.7.118.1:45531/ using document root <none> and password file <none>
I20260929 17:58:12.157785 14492 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:12.157919 14492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:12.162907 14492 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:58:12.165884 14596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:12.168022 14492 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 17:58:12.168236 14492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "38ba528a27114efbae960672fb79404b"
format_stamp: "Formatted at 2026-09-29 17:58:10 on dist-test-slave-r7ph"
server_key: "a8f741fb900a1b17aa365076ea9a4b6a"
server_key_iv: "b5825b46af57991078324efc81285752"
server_key_version: "encryptionkey@0"
I20260929 17:58:12.169157 14492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:12.193272 14492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:12.197866 14492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:12.198132 14492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:12.198882 14492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:12.199878 14492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:12.199951 14492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:12.200038 14492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:12.200085 14492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:12.211803 14492 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:42087
I20260929 17:58:12.211876 14709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:42087 every 8 connection(s)
I20260929 17:58:12.213246 14492 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 17:58:12.213605 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 14492
I20260929 17:58:12.213795 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 13943
I20260929 17:58:12.224472 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:36521
--local_ip_for_outbound_sockets=127.7.118.2
--tserver_master_addrs=127.7.118.62:45353
--webserver_port=37979
--webserver_interface=127.7.118.2
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.2.keytab
--principal=oryx/127.7.118.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.118.62:45353,127.7.118.61:38403
--tserver_master_addrs=127.7.118.62:45353,127.7.118.61:38403,127.7.118.60:45305 with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 17:58:12 dist-test-slave-r7ph krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704692, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM
Sep 29 17:58:12 dist-test-slave-r7ph krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704692, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.61@KRBTEST.COM
Sep 29 17:58:12 dist-test-slave-r7ph krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704692, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.60@KRBTEST.COM
I20260929 17:58:12.233479 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.216122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:36861 (local address 127.7.118.62:45353)
0929 17:58:12.216265 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:12.216269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:12.217920 (+ 1651us) server_negotiation.cc:407] Connection header received
0929 17:58:12.219089 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:12.219092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:12.219153 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:12.219237 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:12.220643 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.221180 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.223033 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.223376 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.223467 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.227133 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:12.227137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:12.227141 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:12.227174 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:12.229987 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:12.230943 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:12.230948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:12.230950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:12.231083 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:12.231618 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:12.231622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:12.231625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:12.231920 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:12.232026 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:12.233093 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0929 17:58:12.233251 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 17:58:12.235606 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.216702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.7.118.1:36861)
0929 17:58:12.217702 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:12.217791 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:12.218841 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:12.219443 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:12.219450 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:12.219864 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:12.220474 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.220490 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.221328 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.221331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.222870 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.222878 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.223692 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.223698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.224029 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.224777 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:12.224799 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:12.226899 (+ 2100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:12.230155 (+ 3256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:12.230164 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:12.230187 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:12.230812 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:12.231218 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:12.231223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:12.231225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:12.231475 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:12.232117 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:12.232128 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:12.232703 (+ 575us) client_negotiation.cc:769] Sending connection context
0929 17:58:12.233689 (+ 986us) client_negotiation.cc:240] Negotiation successful
0929 17:58:12.234161 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":52,"thread_start_us":606,"threads_started":1}
I20260929 17:58:12.235606 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.221815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:45305 (local address 127.7.118.1:52203)
0929 17:58:12.222218 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:12.222233 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:12.222356 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:12.223407 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:12.223413 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:12.223660 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:12.223800 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.223807 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.225791 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.225794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.226731 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.226743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.227431 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.227438 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.227571 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.228524 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:12.228543 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:12.230707 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:12.233424 (+ 2717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:12.233427 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:12.233430 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:12.233718 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:12.234077 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:12.234081 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:12.234083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:12.234176 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:12.234850 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:12.234853 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:12.234956 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 17:58:12.235037 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 17:58:12.235188 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":135,"threads_started":1}
I20260929 17:58:12.236696 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.221917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:52203 (local address 127.7.118.60:45305)
0929 17:58:12.223102 (+ 1185us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:12.223109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:12.223129 (+ 20us) server_negotiation.cc:407] Connection header received
0929 17:58:12.223179 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:12.223185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:12.223248 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:12.223607 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:12.223973 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.225645 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.226904 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.227262 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.227903 (+ 641us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.230943 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:12.230948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:12.230950 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:12.230990 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:12.233292 (+ 2302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:12.233866 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:12.233869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:12.233871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:12.233968 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:12.234320 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:12.234323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:12.234325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:12.234552 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:12.236268 (+ 1716us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:12.236359 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 17:58:12.236516 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1059,"thread_start_us":98,"threads_started":1}
I20260929 17:58:12.237039 14711 heartbeater.cc:347] Connected to a master server at 127.7.118.62:45353
I20260929 17:58:12.237116 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.222803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:35389 (local address 127.7.118.61:38403)
0929 17:58:12.222943 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:12.222949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:12.222966 (+ 17us) server_negotiation.cc:407] Connection header received
0929 17:58:12.223088 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:12.223091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:12.223142 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:12.223191 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:12.224275 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.225056 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.226419 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.226859 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.226957 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.231149 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:12.231156 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:12.231162 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:12.231202 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:12.233761 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:12.234854 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:12.234859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:12.234863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:12.234976 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:12.235418 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:12.235422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:12.235423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:12.235693 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:12.235800 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:12.236791 (+ 991us) server_negotiation.cc:299] Negotiation successful
0929 17:58:12.236935 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 17:58:12.237326 14711 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:12.237844 14711 heartbeater.cc:511] Master 127.7.118.62:45353 requested a full tablet report, sending...
I20260929 17:58:12.238376 14710 heartbeater.cc:347] Connected to a master server at 127.7.118.60:45305
I20260929 17:58:12.238453 14710 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:12.238623 14710 heartbeater.cc:511] Master 127.7.118.60:45305 requested a full tablet report, sending...
I20260929 17:58:12.239215 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.222419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:38403 (local address 127.7.118.1:35389)
0929 17:58:12.222723 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:12.222737 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:12.222828 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:12.223328 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:12.223331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:12.223939 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:12.224125 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.224132 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.225216 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.225222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.226232 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.226273 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.227153 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.227158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.227288 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.228116 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:12.228137 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:12.230887 (+ 2750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:12.234391 (+ 3504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:12.234396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:12.234401 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:12.234694 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:12.235127 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:12.235132 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:12.235136 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:12.235294 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:12.236414 (+ 1120us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:12.236420 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:12.236561 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 17:58:12.238636 (+ 2075us) client_negotiation.cc:240] Negotiation successful
0929 17:58:12.238910 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":4,"thread_start_us":107,"threads_started":1}
I20260929 17:58:12.239401 13747 ts_manager.cc:194] Re-registered known tserver with Master: 38ba528a27114efbae960672fb79404b (127.7.118.1:42087)
I20260929 17:58:12.239964 14712 heartbeater.cc:347] Connected to a master server at 127.7.118.61:38403
I20260929 17:58:12.240097 14712 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:12.240094 14531 ts_manager.cc:194] Registered new tserver with Master: 38ba528a27114efbae960672fb79404b (127.7.118.1:42087)
I20260929 17:58:12.240222 13747 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.118.1@KRBTEST.COM'} at 127.7.118.1:36861
I20260929 17:58:12.240301 14712 heartbeater.cc:511] Master 127.7.118.61:38403 requested a full tablet report, sending...
I20260929 17:58:12.240741 14206 heartbeater.cc:503] Master 127.7.118.62:45353 was elected leader, sending a full tablet report...
I20260929 17:58:12.241770 14348 ts_manager.cc:194] Registered new tserver with Master: 38ba528a27114efbae960672fb79404b (127.7.118.1:42087)
W20260929 17:58:12.370225 14718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:12.370617 14718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:12.370672 14718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:12.375552 14718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:12.375636 14718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:12.375742 14718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 17:58:12.383081 14718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.2.keytab
--principal=oryx/127.7.118.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:36521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=37979
--tserver_master_addrs=127.7.118.62:45353,127.7.118.61:38403,127.7.118.60:45305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:12.384451 14718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:12.385708 14718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:12.398213 14724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:12.398759 14718 server_base.cc:1034] running on GCE node
W20260929 17:58:12.398829 14723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:12.398829 14726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:12.399775 14718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:12.400614 14718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:12.401856 14718 hybrid_clock.cc:648] HybridClock initialized: now 1790704692401805 us; error 70 us; skew 500 ppm
Sep 29 17:58:12 dist-test-slave-r7ph krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704692, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:12.405285 14718 init.cc:377] Logged in from keytab as oryx/127.7.118.2@KRBTEST.COM (short username oryx)
I20260929 17:58:12.406412 14718 webserver.cc:460] Webserver started at http://127.7.118.2:37979/ using document root <none> and password file <none>
I20260929 17:58:12.407071 14718 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:12.407209 14718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:12.413292 14718 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 17:58:12.416924 14733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:12.418964 14718 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 17:58:12.419175 14718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "6d13d7d66af745589d65193cd03d8f52"
format_stamp: "Formatted at 2026-09-29 17:58:10 on dist-test-slave-r7ph"
server_key: "fd0de8744075b5579ab4758f3afcf80d"
server_key_iv: "2f1254bdfab6749095d5a59eb500fc84"
server_key_version: "encryptionkey@0"
I20260929 17:58:12.419833 14718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:12.420060 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.415938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58594 (local address 127.7.118.60:45305)
0929 17:58:12.416452 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:12.416456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:12.416468 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:58:12.416550 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:12.416554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:12.416628 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:12.416741 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:12.417063 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.417773 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.418923 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.419441 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.419571 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.419741 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:12.419821 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 17:58:12.419908 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428}
I20260929 17:58:12.420060 14304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.415793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:45305 (local address 127.0.0.1:58594)
0929 17:58:12.416147 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:12.416172 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:12.416292 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:12.416804 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:12.416807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:12.416830 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:12.416929 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.416934 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.417940 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.417943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.418789 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.418801 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.419584 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.419588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.419679 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.419703 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 17:58:12.419807 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 17:58:12.419900 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260929 17:58:12.420814 14541 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 17:58:12.421269 14573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 796f18bdc8e94ff69ad6374576106f9f. Latest consensus state: current_term: 1 leader_uuid: "796f18bdc8e94ff69ad6374576106f9f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: NON_VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: true } } }
I20260929 17:58:12.421440 14573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [sys.catalog]: This master's current role is: LEARNER
I20260929 17:58:12.441758 14718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:12.446291 14718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:12.446528 14718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:12.447225 14718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:12.448205 14718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:12.448259 14718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:12.448302 14718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:12.448354 14718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:12.458176 14718 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:36521
I20260929 17:58:12.458449 14846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:36521 every 8 connection(s)
I20260929 17:58:12.459537 14718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 17:58:12.465183 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 14718
I20260929 17:58:12.465364 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 14077
I20260929 17:58:12.473589 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:37857
--local_ip_for_outbound_sockets=127.7.118.3
--tserver_master_addrs=127.7.118.62:45353
--webserver_port=36181
--webserver_interface=127.7.118.3
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.3.keytab
--principal=oryx/127.7.118.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.118.62:45353,127.7.118.61:38403
--tserver_master_addrs=127.7.118.62:45353,127.7.118.61:38403,127.7.118.60:45305 with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 17:58:12 dist-test-slave-r7ph krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704692, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.61@KRBTEST.COM
Sep 29 17:58:12 dist-test-slave-r7ph krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704692, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.60@KRBTEST.COM
Sep 29 17:58:12 dist-test-slave-r7ph krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704692, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM
I20260929 17:58:12.480683 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.462582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:54455 (local address 127.7.118.61:38403)
0929 17:58:12.462713 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:12.462720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:12.463775 (+ 1055us) server_negotiation.cc:407] Connection header received
0929 17:58:12.465123 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:12.465127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:12.465189 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:12.465243 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:12.467433 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.467968 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.469562 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.469929 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.470115 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.475316 (+ 5201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:12.475322 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:12.475325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:12.475365 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:12.477372 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:12.478230 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:12.478236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:12.478238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:12.478374 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:12.479160 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:12.479163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:12.479165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:12.479427 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:12.479512 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:12.480375 (+ 863us) server_negotiation.cc:299] Negotiation successful
0929 17:58:12.480505 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 17:58:12.480842 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.461685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:50225 (local address 127.7.118.60:45305)
0929 17:58:12.462451 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:12.462458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:12.463808 (+ 1350us) server_negotiation.cc:407] Connection header received
0929 17:58:12.468694 (+ 4886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:12.468698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:12.468765 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:12.468850 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:12.469462 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.470372 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.471628 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.471985 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.472070 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.475697 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:12.475704 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:12.475709 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:12.475746 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:12.477900 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:12.478597 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:12.478602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:12.478607 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:12.478700 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:12.479200 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:12.479206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:12.479208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:12.479500 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:12.479596 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:12.480411 (+ 815us) server_negotiation.cc:299] Negotiation successful
0929 17:58:12.480605 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657}
I20260929 17:58:12.481899 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.462597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.7.118.2:45183)
0929 17:58:12.463595 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:12.463660 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:12.464843 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:12.466423 (+ 1580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:12.466429 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:12.466670 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:12.468751 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.468770 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.470222 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.470228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.471605 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.471620 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.472642 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.472648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.472776 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.473252 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:12.473267 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:12.475483 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:12.478495 (+ 3012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:12.478498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:12.478500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:12.478810 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:12.479129 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:12.479143 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:12.479145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:12.479232 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:12.479717 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:12.479719 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:12.480063 (+ 344us) client_negotiation.cc:769] Sending connection context
0929 17:58:12.480296 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 17:58:12.480687 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":1966,"spinlock_wait_cycles":18048,"thread_start_us":126,"threads_started":1}
I20260929 17:58:12.481900 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.462286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:45305 (local address 127.7.118.2:50225)
0929 17:58:12.463595 (+ 1309us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:12.463660 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:12.464843 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:12.468972 (+ 4129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:12.468975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:12.469193 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:12.469313 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.469319 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.470509 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.470512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.471467 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.471480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.472253 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.472256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.472382 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.473034 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:12.473051 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:12.475489 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:12.478051 (+ 2562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:12.478057 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:12.478059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:12.478396 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:12.478811 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:12.478817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:12.478822 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:12.479060 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:12.479658 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:12.479661 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:12.480066 (+ 405us) client_negotiation.cc:769] Sending connection context
0929 17:58:12.481140 (+ 1074us) client_negotiation.cc:240] Negotiation successful
0929 17:58:12.481371 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1060,"mutex_wait_us":31,"spinlock_wait_cycles":198016,"thread_start_us":149,"threads_started":1}
I20260929 17:58:12.482440 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.462286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:38403 (local address 127.7.118.2:54455)
0929 17:58:12.463595 (+ 1309us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:12.463660 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:12.464820 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:12.465695 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:12.465705 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:12.466282 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:12.467241 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.467265 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.468134 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.468138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.469428 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.469438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.470411 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.470419 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.470850 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.471649 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:12.471671 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:12.474985 (+ 3314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:12.477541 (+ 2556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:12.477551 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:12.477575 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:12.478083 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:12.478495 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:12.478498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:12.478500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:12.479045 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:12.479592 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:12.479602 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:12.480063 (+ 461us) client_negotiation.cc:769] Sending connection context
0929 17:58:12.481058 (+ 995us) client_negotiation.cc:240] Negotiation successful
0929 17:58:12.481218 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1196,"mutex_wait_us":559,"spinlock_wait_cycles":25216,"thread_start_us":89,"threads_started":1}
I20260929 17:58:12.482808 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.462436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:45183 (local address 127.7.118.62:45353)
0929 17:58:12.462573 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:12.462577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:12.463816 (+ 1239us) server_negotiation.cc:407] Connection header received
0929 17:58:12.465180 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:12.465185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:12.465251 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:12.465329 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:12.468906 (+ 3577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.469991 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.471770 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.472208 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.472313 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.476258 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:12.476264 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:12.476270 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:12.476307 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:12.478379 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:12.478914 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:12.478917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:12.478919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:12.479005 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:12.479347 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:12.479350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:12.479352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:12.479581 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:12.479661 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:12.482500 (+ 2839us) server_negotiation.cc:299] Negotiation successful
0929 17:58:12.482655 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 17:58:12.483966 14849 heartbeater.cc:347] Connected to a master server at 127.7.118.61:38403
I20260929 17:58:12.483994 14848 heartbeater.cc:347] Connected to a master server at 127.7.118.62:45353
I20260929 17:58:12.484122 14847 heartbeater.cc:347] Connected to a master server at 127.7.118.60:45305
I20260929 17:58:12.484251 14848 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:12.484251 14849 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:12.484274 14847 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:12.485009 14849 heartbeater.cc:511] Master 127.7.118.61:38403 requested a full tablet report, sending...
I20260929 17:58:12.485009 14847 heartbeater.cc:511] Master 127.7.118.60:45305 requested a full tablet report, sending...
I20260929 17:58:12.485852 14848 heartbeater.cc:511] Master 127.7.118.62:45353 requested a full tablet report, sending...
I20260929 17:58:12.486559 14348 ts_manager.cc:194] Registered new tserver with Master: 6d13d7d66af745589d65193cd03d8f52 (127.7.118.2:36521)
I20260929 17:58:12.486667 14531 ts_manager.cc:194] Registered new tserver with Master: 6d13d7d66af745589d65193cd03d8f52 (127.7.118.2:36521)
I20260929 17:58:12.487149 13747 ts_manager.cc:194] Re-registered known tserver with Master: 6d13d7d66af745589d65193cd03d8f52 (127.7.118.2:36521)
I20260929 17:58:12.488096 13747 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.118.2@KRBTEST.COM'} at 127.7.118.2:45183
W20260929 17:58:12.600580 14854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:12.600885 14854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:12.600934 14854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:12.605285 14854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:12.605381 14854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:12.605445 14854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 17:58:12.611217 14854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.3.keytab
--principal=oryx/127.7.118.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:37857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=36181
--tserver_master_addrs=127.7.118.62:45353,127.7.118.61:38403,127.7.118.60:45305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:12.612353 14854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:12.613514 14854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:12.624725 14862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:12.624655 14859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:12.624655 14860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:12.625334 14854 server_base.cc:1034] running on GCE node
I20260929 17:58:12.625903 14854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:12.626817 14854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:12.628046 14854 hybrid_clock.cc:648] HybridClock initialized: now 1790704692628030 us; error 54 us; skew 500 ppm
Sep 29 17:58:12 dist-test-slave-r7ph krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704692, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:12.631959 14854 init.cc:377] Logged in from keytab as oryx/127.7.118.3@KRBTEST.COM (short username oryx)
I20260929 17:58:12.633050 14854 webserver.cc:460] Webserver started at http://127.7.118.3:36181/ using document root <none> and password file <none>
I20260929 17:58:12.633701 14854 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:12.633810 14854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:12.638447 14854 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:58:12.640895 14869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:12.642462 14854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:58:12.642632 14854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "0dd99fb321114c759e2cf85a6fff9ee7"
format_stamp: "Formatted at 2026-09-29 17:58:11 on dist-test-slave-r7ph"
server_key: "5e9dada01b8be6cd3425403435e0319d"
server_key_iv: "4b55c85868317ef6da470a937262b5fb"
server_key_version: "encryptionkey@0"
I20260929 17:58:12.643317 14854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:12.676678 14854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:12.680868 14854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:12.681079 14854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:12.681764 14854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:12.682894 14854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:12.682963 14854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:12.683034 14854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:12.683085 14854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:12.693006 14854 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:37857
I20260929 17:58:12.693068 14982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:37857 every 8 connection(s)
I20260929 17:58:12.694130 14854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 17:58:12.702468 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 14854
Sep 29 17:58:12 dist-test-slave-r7ph krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704692, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.61@KRBTEST.COM
Sep 29 17:58:12 dist-test-slave-r7ph krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704692, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.60@KRBTEST.COM
Sep 29 17:58:12 dist-test-slave-r7ph krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704692, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM
I20260929 17:58:12.716040 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.696184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:47351 (local address 127.7.118.62:45353)
0929 17:58:12.696374 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:12.696379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:12.697769 (+ 1390us) server_negotiation.cc:407] Connection header received
0929 17:58:12.698683 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:12.698686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:12.698741 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:12.698795 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:12.700403 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.700965 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.702990 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.703352 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.703438 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.709812 (+ 6374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:12.709815 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:12.709818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:12.709851 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:12.712925 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:12.713645 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:12.713651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:12.713654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:12.713750 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:12.714238 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:12.714241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:12.714244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:12.714521 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:12.714614 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:12.715548 (+ 934us) server_negotiation.cc:299] Negotiation successful
0929 17:58:12.715861 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 17:58:12.717037 14987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.696642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:38403 (local address 127.7.118.3:46745)
0929 17:58:12.697217 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:12.697277 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:12.698433 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:12.699021 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:12.699029 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:12.699559 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:12.700202 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.700216 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.701442 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.701445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.702820 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.702833 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.703681 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.703685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.704446 (+ 761us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.705091 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:12.705112 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:12.710424 (+ 5312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:12.713980 (+ 3556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:12.713983 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:12.713985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:12.714230 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:12.714567 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:12.714571 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:12.714577 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:12.714667 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:12.715115 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:12.715118 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:12.715196 (+ 78us) client_negotiation.cc:769] Sending connection context
0929 17:58:12.715297 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 17:58:12.715972 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":294,"spinlock_wait_cycles":13568,"thread_start_us":171,"threads_started":1}
I20260929 17:58:12.717362 14989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.697785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:45305 (local address 127.7.118.3:32871)
0929 17:58:12.698096 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:12.698108 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:12.698433 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:12.699019 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:12.699027 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:12.699539 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:12.700397 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.700412 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.701520 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.701523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.702833 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.702845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.703838 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.703841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.704004 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.704842 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:12.704867 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:12.708764 (+ 3897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:12.714623 (+ 5859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:12.714627 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:12.714629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:12.714891 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:12.715253 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:12.715256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:12.715258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:12.715341 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:12.715836 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:12.715842 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:12.715939 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 17:58:12.716035 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 17:58:12.716209 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":236,"spinlock_wait_cycles":83328,"thread_start_us":116,"threads_started":1}
I20260929 17:58:12.717770 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.696218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:46745 (local address 127.7.118.61:38403)
0929 17:58:12.696542 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:12.696547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:12.697409 (+ 862us) server_negotiation.cc:407] Connection header received
0929 17:58:12.698682 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:12.698686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:12.698741 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:12.698795 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:12.700439 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.701323 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.702990 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.703352 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.703438 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.710619 (+ 7181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:12.710623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:12.710626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:12.710656 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:12.713866 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:12.714366 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:12.714369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:12.714371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:12.714449 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:12.714768 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:12.714771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:12.714773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:12.714966 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:12.715072 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:12.717510 (+ 2438us) server_negotiation.cc:299] Negotiation successful
0929 17:58:12.717653 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260929 17:58:12.718024 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.697883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:32871 (local address 127.7.118.60:45305)
0929 17:58:12.698051 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:12.698055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:12.698186 (+ 131us) server_negotiation.cc:407] Connection header received
0929 17:58:12.698649 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:12.698660 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:12.698723 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:12.698821 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:12.700572 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.701405 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.703003 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.703440 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.703553 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.708976 (+ 5423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:12.708980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:12.708983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:12.709012 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:12.714471 (+ 5459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:12.715030 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:12.715034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:12.715036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:12.715148 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:12.715452 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:12.715455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:12.715457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:12.715697 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:12.715793 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:12.717711 (+ 1918us) server_negotiation.cc:299] Negotiation successful
0929 17:58:12.717864 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 17:58:12.718246 14988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.696643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.7.118.3:47351)
0929 17:58:12.697668 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:12.697681 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:12.698433 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:12.699030 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:12.699033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:12.699565 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:12.700194 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.700216 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.701124 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.701128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.702831 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.702836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.703681 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.703685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.704003 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.704964 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:12.704977 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:12.709662 (+ 4685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:12.713066 (+ 3404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:12.713080 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:12.713106 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:12.713511 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:12.713861 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:12.713864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:12.713866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:12.714111 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:12.714667 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:12.714675 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:12.715081 (+ 406us) client_negotiation.cc:769] Sending connection context
0929 17:58:12.715297 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 17:58:12.718065 (+ 2768us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"mutex_wait_us":2750,"spinlock_wait_cycles":32768,"thread_start_us":163,"threads_started":1}
I20260929 17:58:12.718783 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.705461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40728 (local address 127.7.118.62:45353)
0929 17:58:12.706223 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:12.706228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:12.706242 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:58:12.706314 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:12.706317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:12.706374 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:12.706476 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:12.708308 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.709054 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.711051 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.711436 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.711560 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.712939 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:12.712944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:12.712950 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:12.712979 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:12.715048 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:12.715581 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:12.715584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:12.715586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:12.715687 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:12.716011 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:12.716017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:12.716020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:12.716542 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:12.716693 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:12.717711 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0929 17:58:12.718241 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":84,"threads_started":1}
I20260929 17:58:12.718909 14983 heartbeater.cc:347] Connected to a master server at 127.7.118.60:45305
I20260929 17:58:12.719209 14983 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:12.719889 14983 heartbeater.cc:511] Master 127.7.118.60:45305 requested a full tablet report, sending...
I20260929 17:58:12.720552 14985 heartbeater.cc:347] Connected to a master server at 127.7.118.61:38403
I20260929 17:58:12.720639 14985 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:12.720851 14985 heartbeater.cc:511] Master 127.7.118.61:38403 requested a full tablet report, sending...
I20260929 17:58:12.721411 14531 ts_manager.cc:194] Registered new tserver with Master: 0dd99fb321114c759e2cf85a6fff9ee7 (127.7.118.3:37857)
I20260929 17:58:12.721508 14984 heartbeater.cc:347] Connected to a master server at 127.7.118.62:45353
I20260929 17:58:12.721582 14984 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:12.721740 14984 heartbeater.cc:511] Master 127.7.118.62:45353 requested a full tablet report, sending...
I20260929 17:58:12.722397 14995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.705306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58458 (local address 127.7.118.61:38403)
0929 17:58:12.706473 (+ 1167us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:12.706477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:12.706720 (+ 243us) server_negotiation.cc:407] Connection header received
0929 17:58:12.706923 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:12.706926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:12.706996 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:12.707270 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:12.707952 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.708857 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.710414 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.710836 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.711357 (+ 521us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.712778 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:12.712782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:12.712785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:12.712816 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:12.719936 (+ 7120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:12.720472 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:12.720476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:12.720478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:12.720603 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:12.720996 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:12.720999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:12.721001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:12.721239 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:12.721985 (+ 746us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:12.722065 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 17:58:12.722200 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1087,"thread_start_us":83,"threads_started":1}
I20260929 17:58:12.722395 13747 ts_manager.cc:194] Re-registered known tserver with Master: 0dd99fb321114c759e2cf85a6fff9ee7 (127.7.118.3:37857)
I20260929 17:58:12.723107 13747 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.118.3@KRBTEST.COM'} at 127.7.118.3:47351
I20260929 17:58:12.724393 14348 ts_manager.cc:194] Registered new tserver with Master: 0dd99fb321114c759e2cf85a6fff9ee7 (127.7.118.3:37857)
I20260929 17:58:12.725371 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.704963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58598 (local address 127.7.118.60:45305)
0929 17:58:12.706160 (+ 1197us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:12.706165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:12.707205 (+ 1040us) server_negotiation.cc:407] Connection header received
0929 17:58:12.707397 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:12.707400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:12.707449 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:12.707517 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:12.708353 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.709081 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.719168 (+ 10087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.719503 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.719566 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.721436 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:12.721440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:12.721443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:12.721468 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:12.723456 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:12.723989 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:12.723992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:12.723994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:12.724086 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:12.724408 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:12.724412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:12.724413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:12.724617 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:12.724724 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:12.725098 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 17:58:12.725228 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1102,"thread_start_us":117,"threads_started":1}
I20260929 17:58:12.728904 13747 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:58:12.732280 13747 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:58:12.735248 14495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "796f18bdc8e94ff69ad6374576106f9f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: NON_VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: true } } }
I20260929 17:58:12.735417 14495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:12.739341 15003 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f: attempting to promote NON_VOTER 9d44b2b97944484f88c630984887cf55 to VOTER
I20260929 17:58:12.739861 15003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: false } }
I20260929 17:58:12.743917 14358 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 1 FOLLOWER]: Refusing update from remote peer 796f18bdc8e94ff69ad6374576106f9f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 17:58:12.744414 14495 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260929 17:58:12.744557 14541 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 1 LEARNER]: Refusing update from remote peer 796f18bdc8e94ff69ad6374576106f9f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 17:58:12.745236 14495 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260929 17:58:12.746742 14495 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 9d44b2b97944484f88c630984887cf55 (127.7.118.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: false } } }
I20260929 17:58:12.747210 14358 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 9d44b2b97944484f88c630984887cf55 (127.7.118.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: false } } }
I20260929 17:58:12.747843 14541 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 9d44b2b97944484f88c630984887cf55 (127.7.118.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: false } } }
I20260929 17:58:12.748754 14495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "796f18bdc8e94ff69ad6374576106f9f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: false } } }
I20260929 17:58:12.748889 14495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:12.749017 14495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "796f18bdc8e94ff69ad6374576106f9f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: false } } }
I20260929 17:58:12.749101 14495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:12.749258 14495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "796f18bdc8e94ff69ad6374576106f9f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: false } } }
I20260929 17:58:12.749341 14495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:12.749606 14573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "796f18bdc8e94ff69ad6374576106f9f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: false } } }
I20260929 17:58:12.749763 14573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:58:12.750451 15011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "796f18bdc8e94ff69ad6374576106f9f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: false } } }
I20260929 17:58:12.750615 15011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:58:12.757638 14304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.751902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:36521 (local address 127.0.0.1:33056)
0929 17:58:12.752058 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:12.752075 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:12.752173 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:12.753323 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:12.753326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:12.753348 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:12.753463 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.753469 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.755140 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.755144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.756164 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.756174 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.757224 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.757227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.757319 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.757352 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 17:58:12.757401 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 17:58:12.757479 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":25}
I20260929 17:58:12.758543 15012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.752102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33056 (local address 127.7.118.2:36521)
0929 17:58:12.752717 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:12.752731 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:12.752762 (+ 31us) server_negotiation.cc:407] Connection header received
0929 17:58:12.752861 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:12.752868 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:12.753134 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:12.753637 (+ 503us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:12.753787 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.755018 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.756296 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.757071 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.757962 (+ 891us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.758169 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:12.758231 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 17:58:12.758389 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":186,"threads_started":1}
I20260929 17:58:12.758641 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.752415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:42087 (local address 127.0.0.1:34534)
0929 17:58:12.752920 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:12.752942 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:12.753075 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:12.753940 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:12.753943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:12.753962 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:12.754066 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.754074 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.755666 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.755670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.756909 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.756920 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.758084 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.758087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.758230 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.758313 (+ 83us) client_negotiation.cc:769] Sending connection context
0929 17:58:12.758381 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 17:58:12.758466 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":53,"thread_start_us":159,"threads_started":1}
I20260929 17:58:12.758765 15015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.752608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34534 (local address 127.7.118.1:42087)
0929 17:58:12.753366 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:12.753372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:12.753394 (+ 22us) server_negotiation.cc:407] Connection header received
0929 17:58:12.753467 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:12.753472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:12.753704 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:12.754360 (+ 656us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:12.754478 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.755502 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.757062 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.757930 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.758310 (+ 380us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.758460 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:12.758515 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 17:58:12.758617 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":59,"threads_started":1}
I20260929 17:58:12.758965 15014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.752524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:37857 (local address 127.0.0.1:55904)
0929 17:58:12.753922 (+ 1398us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:12.753935 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:12.754030 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:12.754592 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:12.754595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:12.754614 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:12.754714 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.754720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.755861 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.755864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.757025 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.757033 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.758068 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.758072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.758211 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.758296 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 17:58:12.758368 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 17:58:12.758441 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1339,"mutex_wait_us":52,"thread_start_us":128,"threads_started":1}
I20260929 17:58:12.759356 15016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.752963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55904 (local address 127.7.118.3:37857)
0929 17:58:12.753299 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:12.753312 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:12.754129 (+ 817us) server_negotiation.cc:407] Connection header received
0929 17:58:12.754211 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:12.754217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:12.754440 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:12.754601 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:12.754832 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.755735 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.757156 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.757952 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.758060 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.758217 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:12.758574 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 17:58:12.759216 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":101,"threads_started":1}
I20260929 17:58:12.761698 14644 tablet_service.cc:1467] Processing CreateTablet for tablet 4260a7533d0f4c16b822235a6b9edb22 (DEFAULT_TABLE table=test-table [id=653504536d0b4e69a5e8f8bc4e2c88b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:58:12.761696 14781 tablet_service.cc:1467] Processing CreateTablet for tablet 4260a7533d0f4c16b822235a6b9edb22 (DEFAULT_TABLE table=test-table [id=653504536d0b4e69a5e8f8bc4e2c88b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:58:12.762315 14917 tablet_service.cc:1467] Processing CreateTablet for tablet 4260a7533d0f4c16b822235a6b9edb22 (DEFAULT_TABLE table=test-table [id=653504536d0b4e69a5e8f8bc4e2c88b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:58:12.762787 14781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4260a7533d0f4c16b822235a6b9edb22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:12.762775 14644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4260a7533d0f4c16b822235a6b9edb22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:12.763316 14917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4260a7533d0f4c16b822235a6b9edb22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:12.770028 15017 tablet_bootstrap.cc:492] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b: Bootstrap starting.
I20260929 17:58:12.771595 15018 tablet_bootstrap.cc:492] T 4260a7533d0f4c16b822235a6b9edb22 P 6d13d7d66af745589d65193cd03d8f52: Bootstrap starting.
I20260929 17:58:12.773196 15017 tablet_bootstrap.cc:654] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:12.774082 15018 tablet_bootstrap.cc:654] T 4260a7533d0f4c16b822235a6b9edb22 P 6d13d7d66af745589d65193cd03d8f52: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:12.774737 15017 log.cc:824] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:12.775187 15019 tablet_bootstrap.cc:492] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7: Bootstrap starting.
I20260929 17:58:12.775820 15018 log.cc:824] T 4260a7533d0f4c16b822235a6b9edb22 P 6d13d7d66af745589d65193cd03d8f52: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:12.777284 15017 tablet_bootstrap.cc:492] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b: No bootstrap required, opened a new log
I20260929 17:58:12.777530 15017 ts_tablet_manager.cc:1397] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 17:58:12.777695 15019 tablet_bootstrap.cc:654] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:12.778146 15018 tablet_bootstrap.cc:492] T 4260a7533d0f4c16b822235a6b9edb22 P 6d13d7d66af745589d65193cd03d8f52: No bootstrap required, opened a new log
I20260929 17:58:12.778401 15018 ts_tablet_manager.cc:1397] T 4260a7533d0f4c16b822235a6b9edb22 P 6d13d7d66af745589d65193cd03d8f52: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 17:58:12.778625 15019 log.cc:824] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:12.781026 15019 tablet_bootstrap.cc:492] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7: No bootstrap required, opened a new log
I20260929 17:58:12.781241 15019 ts_tablet_manager.cc:1397] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 17:58:12.782096 15017 raft_consensus.cc:348] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d13d7d66af745589d65193cd03d8f52" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 36521 } } peers { permanent_uuid: "38ba528a27114efbae960672fb79404b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42087 } } peers { permanent_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37857 } }
I20260929 17:58:12.782444 15017 raft_consensus.cc:374] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:12.782502 15017 raft_consensus.cc:729] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ba528a27114efbae960672fb79404b, State: Initialized, Role: FOLLOWER
I20260929 17:58:12.782640 15018 raft_consensus.cc:348] T 4260a7533d0f4c16b822235a6b9edb22 P 6d13d7d66af745589d65193cd03d8f52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d13d7d66af745589d65193cd03d8f52" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 36521 } } peers { permanent_uuid: "38ba528a27114efbae960672fb79404b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42087 } } peers { permanent_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37857 } }
I20260929 17:58:12.782913 15018 raft_consensus.cc:374] T 4260a7533d0f4c16b822235a6b9edb22 P 6d13d7d66af745589d65193cd03d8f52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:12.782968 15018 raft_consensus.cc:729] T 4260a7533d0f4c16b822235a6b9edb22 P 6d13d7d66af745589d65193cd03d8f52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d13d7d66af745589d65193cd03d8f52, State: Initialized, Role: FOLLOWER
I20260929 17:58:12.783041 15017 consensus_queue.cc:260] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d13d7d66af745589d65193cd03d8f52" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 36521 } } peers { permanent_uuid: "38ba528a27114efbae960672fb79404b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42087 } } peers { permanent_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37857 } }
I20260929 17:58:12.783427 15018 consensus_queue.cc:260] T 4260a7533d0f4c16b822235a6b9edb22 P 6d13d7d66af745589d65193cd03d8f52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d13d7d66af745589d65193cd03d8f52" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 36521 } } peers { permanent_uuid: "38ba528a27114efbae960672fb79404b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42087 } } peers { permanent_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37857 } }
I20260929 17:58:12.784152 15017 ts_tablet_manager.cc:1428] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 17:58:12.784387 14711 heartbeater.cc:503] Master 127.7.118.62:45353 was elected leader, sending a full tablet report...
I20260929 17:58:12.784471 15018 ts_tablet_manager.cc:1428] T 4260a7533d0f4c16b822235a6b9edb22 P 6d13d7d66af745589d65193cd03d8f52: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 17:58:12.786156 15019 raft_consensus.cc:348] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d13d7d66af745589d65193cd03d8f52" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 36521 } } peers { permanent_uuid: "38ba528a27114efbae960672fb79404b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42087 } } peers { permanent_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37857 } }
I20260929 17:58:12.786473 15019 raft_consensus.cc:374] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:12.786530 15019 raft_consensus.cc:729] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dd99fb321114c759e2cf85a6fff9ee7, State: Initialized, Role: FOLLOWER
I20260929 17:58:12.787024 15019 consensus_queue.cc:260] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d13d7d66af745589d65193cd03d8f52" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 36521 } } peers { permanent_uuid: "38ba528a27114efbae960672fb79404b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42087 } } peers { permanent_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37857 } }
I20260929 17:58:12.788141 15019 ts_tablet_manager.cc:1428] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 17:58:12.789448 14848 heartbeater.cc:503] Master 127.7.118.62:45353 was elected leader, sending a full tablet report...
I20260929 17:58:12.789958 14984 heartbeater.cc:503] Master 127.7.118.62:45353 was elected leader, sending a full tablet report...
I20260929 17:58:12.918020 15025 raft_consensus.cc:482] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:58:12.918327 15025 raft_consensus.cc:504] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d13d7d66af745589d65193cd03d8f52" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 36521 } } peers { permanent_uuid: "38ba528a27114efbae960672fb79404b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42087 } } peers { permanent_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37857 } }
I20260929 17:58:12.919776 15025 leader_election.cc:288] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d13d7d66af745589d65193cd03d8f52 (127.7.118.2:36521), 38ba528a27114efbae960672fb79404b (127.7.118.1:42087)
I20260929 17:58:12.920584 15023 raft_consensus.cc:482] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:58:12.920789 15023 raft_consensus.cc:504] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d13d7d66af745589d65193cd03d8f52" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 36521 } } peers { permanent_uuid: "38ba528a27114efbae960672fb79404b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42087 } } peers { permanent_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37857 } }
I20260929 17:58:12.922756 15023 leader_election.cc:288] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d13d7d66af745589d65193cd03d8f52 (127.7.118.2:36521), 0dd99fb321114c759e2cf85a6fff9ee7 (127.7.118.3:37857)
I20260929 17:58:12.924813 14989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.920066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:42087 (local address 127.7.118.3:53923)
0929 17:58:12.920307 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:12.920319 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:12.920379 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:12.920827 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:12.920830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:12.920848 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:12.920964 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.920971 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.921936 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.921939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.923201 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.923211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.924240 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.924245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.924457 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.924497 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 17:58:12.924566 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 17:58:12.924639 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":223}
I20260929 17:58:12.924851 15012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.920221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:52215 (local address 127.7.118.2:36521)
0929 17:58:12.920351 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:12.920356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:12.920404 (+ 48us) server_negotiation.cc:407] Connection header received
0929 17:58:12.920464 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:12.920468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:12.920530 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:12.920623 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:12.920923 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.921726 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.923361 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.924007 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.924102 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.924182 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:12.924614 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 17:58:12.924703 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":51}
I20260929 17:58:12.925138 14988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.920050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:36521 (local address 127.7.118.3:52215)
0929 17:58:12.920205 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:12.920223 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:12.920331 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:12.920640 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:12.920643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:12.920666 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:12.920796 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.920805 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.921864 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.921868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.923227 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.923240 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.924256 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.924265 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.924395 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.924419 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 17:58:12.924508 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 17:58:12.925021 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":450}
I20260929 17:58:12.925326 15015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.920389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:53923 (local address 127.7.118.1:42087)
0929 17:58:12.920507 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:12.920512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:12.920525 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:58:12.920580 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:12.920582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:12.920633 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:12.920710 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:12.921093 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.921811 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.923334 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.924006 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.924100 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.924366 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:12.925080 (+ 714us) server_negotiation.cc:299] Negotiation successful
0929 17:58:12.925156 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":171,"server-negotiator.queue_time_us":34}
I20260929 17:58:12.925940 14664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4260a7533d0f4c16b822235a6b9edb22" candidate_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ba528a27114efbae960672fb79404b" is_pre_election: true
I20260929 17:58:12.926203 14664 raft_consensus.cc:2454] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dd99fb321114c759e2cf85a6fff9ee7 in term 0.
I20260929 17:58:12.926486 14801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4260a7533d0f4c16b822235a6b9edb22" candidate_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d13d7d66af745589d65193cd03d8f52" is_pre_election: true
I20260929 17:58:12.926769 14873 leader_election.cc:302] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dd99fb321114c759e2cf85a6fff9ee7, 38ba528a27114efbae960672fb79404b; no voters:
I20260929 17:58:12.926826 14801 raft_consensus.cc:2454] T 4260a7533d0f4c16b822235a6b9edb22 P 6d13d7d66af745589d65193cd03d8f52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dd99fb321114c759e2cf85a6fff9ee7 in term 0.
I20260929 17:58:12.927127 15025 raft_consensus.cc:2781] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:58:12.927251 15025 raft_consensus.cc:482] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:58:12.927312 15025 raft_consensus.cc:3037] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:12.927742 15028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.922555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:37477 (local address 127.7.118.2:36521)
0929 17:58:12.922999 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:12.923045 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:12.923061 (+ 16us) server_negotiation.cc:407] Connection header received
0929 17:58:12.923110 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:12.923113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:12.923160 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:12.923245 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:12.923850 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.924640 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.926326 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.926971 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.927081 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.927182 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:12.927508 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 17:58:12.927586 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":84,"threads_started":1}
I20260929 17:58:12.927771 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.922142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:36521 (local address 127.7.118.1:37477)
0929 17:58:12.922918 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:12.922934 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:12.923042 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:12.923384 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:12.923387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:12.923409 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:12.923640 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.923656 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.924779 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.924782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.926149 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.926160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.927247 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.927251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.927372 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.927401 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 17:58:12.927485 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 17:58:12.927585 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":94,"thread_start_us":148,"threads_started":1}
I20260929 17:58:12.928264 14801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4260a7533d0f4c16b822235a6b9edb22" candidate_uuid: "38ba528a27114efbae960672fb79404b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d13d7d66af745589d65193cd03d8f52" is_pre_election: true
I20260929 17:58:12.928404 14801 raft_consensus.cc:2454] T 4260a7533d0f4c16b822235a6b9edb22 P 6d13d7d66af745589d65193cd03d8f52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38ba528a27114efbae960672fb79404b in term 0.
I20260929 17:58:12.928802 14598 leader_election.cc:302] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ba528a27114efbae960672fb79404b, 6d13d7d66af745589d65193cd03d8f52; no voters:
I20260929 17:58:12.928853 15025 raft_consensus.cc:504] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d13d7d66af745589d65193cd03d8f52" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 36521 } } peers { permanent_uuid: "38ba528a27114efbae960672fb79404b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42087 } } peers { permanent_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37857 } }
I20260929 17:58:12.929102 15016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.922572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:46045 (local address 127.7.118.3:37857)
0929 17:58:12.922705 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:12.922709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:12.923611 (+ 902us) server_negotiation.cc:407] Connection header received
0929 17:58:12.923815 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:12.923818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:12.923878 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:12.923924 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:12.924410 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.925157 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.927785 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.928424 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.928507 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.928581 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:12.928907 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 17:58:12.928989 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":43}
I20260929 17:58:12.929167 15023 raft_consensus.cc:2781] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:58:12.929255 15023 raft_consensus.cc:482] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:58:12.929276 15025 leader_election.cc:288] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7 [CANDIDATE]: Term 1 election: Requested vote from peers 6d13d7d66af745589d65193cd03d8f52 (127.7.118.2:36521), 38ba528a27114efbae960672fb79404b (127.7.118.1:42087)
I20260929 17:58:12.929322 15023 raft_consensus.cc:3037] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:12.929410 15026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.922158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:37857 (local address 127.7.118.1:46045)
0929 17:58:12.923530 (+ 1372us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:12.923544 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:12.923705 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:12.924077 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:12.924080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:12.924095 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:12.924275 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.924284 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.925537 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.925540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.927640 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:12.927650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.928589 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.928593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:12.928759 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.928796 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 17:58:12.928885 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 17:58:12.929044 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1298,"mutex_wait_us":2,"thread_start_us":144,"threads_started":1}
I20260929 17:58:12.929742 14801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4260a7533d0f4c16b822235a6b9edb22" candidate_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d13d7d66af745589d65193cd03d8f52"
I20260929 17:58:12.929875 14801 raft_consensus.cc:3037] T 4260a7533d0f4c16b822235a6b9edb22 P 6d13d7d66af745589d65193cd03d8f52 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:12.930163 14937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4260a7533d0f4c16b822235a6b9edb22" candidate_uuid: "38ba528a27114efbae960672fb79404b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" is_pre_election: true
I20260929 17:58:12.930459 14937 raft_consensus.cc:2379] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38ba528a27114efbae960672fb79404b in current term 1: Already voted for candidate 0dd99fb321114c759e2cf85a6fff9ee7 in this term.
I20260929 17:58:12.930454 14664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4260a7533d0f4c16b822235a6b9edb22" candidate_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ba528a27114efbae960672fb79404b"
I20260929 17:58:12.930851 15023 raft_consensus.cc:504] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d13d7d66af745589d65193cd03d8f52" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 36521 } } peers { permanent_uuid: "38ba528a27114efbae960672fb79404b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42087 } } peers { permanent_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37857 } }
I20260929 17:58:12.931133 14664 raft_consensus.cc:2379] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0dd99fb321114c759e2cf85a6fff9ee7 in current term 1: Already voted for candidate 38ba528a27114efbae960672fb79404b in this term.
I20260929 17:58:12.931258 15023 leader_election.cc:288] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b [CANDIDATE]: Term 1 election: Requested vote from peers 6d13d7d66af745589d65193cd03d8f52 (127.7.118.2:36521), 0dd99fb321114c759e2cf85a6fff9ee7 (127.7.118.3:37857)
I20260929 17:58:12.931311 14801 raft_consensus.cc:2454] T 4260a7533d0f4c16b822235a6b9edb22 P 6d13d7d66af745589d65193cd03d8f52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dd99fb321114c759e2cf85a6fff9ee7 in term 1.
I20260929 17:58:12.931615 14871 leader_election.cc:302] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dd99fb321114c759e2cf85a6fff9ee7, 6d13d7d66af745589d65193cd03d8f52; no voters:
I20260929 17:58:12.931703 14937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4260a7533d0f4c16b822235a6b9edb22" candidate_uuid: "38ba528a27114efbae960672fb79404b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dd99fb321114c759e2cf85a6fff9ee7"
I20260929 17:58:12.931839 14937 raft_consensus.cc:2379] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 38ba528a27114efbae960672fb79404b in current term 1: Already voted for candidate 0dd99fb321114c759e2cf85a6fff9ee7 in this term.
I20260929 17:58:12.931890 15025 raft_consensus.cc:2781] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:58:12.931936 14801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4260a7533d0f4c16b822235a6b9edb22" candidate_uuid: "38ba528a27114efbae960672fb79404b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d13d7d66af745589d65193cd03d8f52"
I20260929 17:58:12.932150 15025 raft_consensus.cc:686] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7 [term 1 LEADER]: Becoming Leader. State: Replica: 0dd99fb321114c759e2cf85a6fff9ee7, State: Running, Role: LEADER
I20260929 17:58:12.932171 14801 raft_consensus.cc:2379] T 4260a7533d0f4c16b822235a6b9edb22 P 6d13d7d66af745589d65193cd03d8f52 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 38ba528a27114efbae960672fb79404b in current term 1: Already voted for candidate 0dd99fb321114c759e2cf85a6fff9ee7 in this term.
I20260929 17:58:12.932644 14598 leader_election.cc:302] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38ba528a27114efbae960672fb79404b; no voters: 0dd99fb321114c759e2cf85a6fff9ee7, 6d13d7d66af745589d65193cd03d8f52
I20260929 17:58:12.932689 15025 consensus_queue.cc:237] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d13d7d66af745589d65193cd03d8f52" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 36521 } } peers { permanent_uuid: "38ba528a27114efbae960672fb79404b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42087 } } peers { permanent_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37857 } }
I20260929 17:58:12.932873 15023 raft_consensus.cc:2726] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 17:58:12.936359 13743 catalog_manager.cc:5495] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dd99fb321114c759e2cf85a6fff9ee7 (127.7.118.3). New cstate: current_term: 1 leader_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d13d7d66af745589d65193cd03d8f52" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 36521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38ba528a27114efbae960672fb79404b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37857 } health_report { overall_health: HEALTHY } } }
W20260929 17:58:12.945654 14986 tablet.cc:2369] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:58:12.962448 14850 tablet.cc:2369] T 4260a7533d0f4c16b822235a6b9edb22 P 6d13d7d66af745589d65193cd03d8f52: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:58:12.973026 14713 tablet.cc:2369] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:58:12.996592 15016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:12.992188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55916 (local address 127.7.118.3:37857)
0929 17:58:12.992337 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:12.992342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:12.992435 (+ 93us) server_negotiation.cc:407] Connection header received
0929 17:58:12.992643 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:12.992646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:12.992725 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:12.992840 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:58:12.993096 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.993611 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.994731 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:12.994979 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:12.995094 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:12.995451 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:58:12.996120 (+ 669us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:58:12.996308 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:12.996356 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 17:58:12.996424 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 17:58:13.003726 14801 raft_consensus.cc:1261] T 4260a7533d0f4c16b822235a6b9edb22 P 6d13d7d66af745589d65193cd03d8f52 [term 1 FOLLOWER]: Refusing update from remote peer 0dd99fb321114c759e2cf85a6fff9ee7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:58:13.003894 14664 raft_consensus.cc:1261] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b [term 1 FOLLOWER]: Refusing update from remote peer 0dd99fb321114c759e2cf85a6fff9ee7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:58:13.004523 15025 consensus_queue.cc:1035] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d13d7d66af745589d65193cd03d8f52" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 36521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:58:13.004840 15029 consensus_queue.cc:1035] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38ba528a27114efbae960672fb79404b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:58:13.028635 13747 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:58:13.028820 13747 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:58:13.035290 13747 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 796f18bdc8e94ff69ad6374576106f9f: Sending DeleteTablet for 3 replicas of tablet 4260a7533d0f4c16b822235a6b9edb22
I20260929 17:58:13.036309 14781 tablet_service.cc:1514] Processing DeleteTablet for tablet 4260a7533d0f4c16b822235a6b9edb22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:58:13 UTC) from {username='oryx', principal='oryx/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:33056
I20260929 17:58:13.036283 14917 tablet_service.cc:1514] Processing DeleteTablet for tablet 4260a7533d0f4c16b822235a6b9edb22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:58:13 UTC) from {username='oryx', principal='oryx/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:55904
I20260929 17:58:13.036319 14644 tablet_service.cc:1514] Processing DeleteTablet for tablet 4260a7533d0f4c16b822235a6b9edb22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:58:13 UTC) from {username='oryx', principal='oryx/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:34534
I20260929 17:58:13.037141 15044 tablet_replica.cc:330] T 4260a7533d0f4c16b822235a6b9edb22 P 6d13d7d66af745589d65193cd03d8f52: stopping tablet replica
I20260929 17:58:13.037364 15042 tablet_replica.cc:330] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b: stopping tablet replica
I20260929 17:58:13.037554 15044 raft_consensus.cc:2229] T 4260a7533d0f4c16b822235a6b9edb22 P 6d13d7d66af745589d65193cd03d8f52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:58:13.037941 15044 raft_consensus.cc:2258] T 4260a7533d0f4c16b822235a6b9edb22 P 6d13d7d66af745589d65193cd03d8f52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:58:13.037931 15042 raft_consensus.cc:2229] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:58:13.038416 15042 raft_consensus.cc:2258] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:58:13.040221 15042 ts_tablet_manager.cc:1905] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:58:13.040539 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 13715
I20260929 17:58:13.041847 15044 ts_tablet_manager.cc:1905] T 4260a7533d0f4c16b822235a6b9edb22 P 6d13d7d66af745589d65193cd03d8f52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:58:13.044409 15042 ts_tablet_manager.cc:1918] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:58:13.044655 15042 log.cc:1196] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-0/wal/wals/4260a7533d0f4c16b822235a6b9edb22
I20260929 17:58:13.044685 15044 ts_tablet_manager.cc:1918] T 4260a7533d0f4c16b822235a6b9edb22 P 6d13d7d66af745589d65193cd03d8f52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:58:13.044801 15044 log.cc:1196] T 4260a7533d0f4c16b822235a6b9edb22 P 6d13d7d66af745589d65193cd03d8f52: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/wal/wals/4260a7533d0f4c16b822235a6b9edb22
I20260929 17:58:13.045127 15044 ts_tablet_manager.cc:1939] T 4260a7533d0f4c16b822235a6b9edb22 P 6d13d7d66af745589d65193cd03d8f52: Deleting consensus metadata
I20260929 17:58:13.045282 15042 ts_tablet_manager.cc:1939] T 4260a7533d0f4c16b822235a6b9edb22 P 38ba528a27114efbae960672fb79404b: Deleting consensus metadata
W20260929 17:58:13.049790 14599 connection.cc:535] server connection from 127.0.0.1:34534 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 17:58:13.050333 15043 tablet_replica.cc:330] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7: stopping tablet replica
I20260929 17:58:13.050769 15043 raft_consensus.cc:2229] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:58:13.051280 15043 raft_consensus.cc:2258] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 17:58:13.052789 14735 connection.cc:441] server connection from 127.0.0.1:33056 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33056 (request call id 1) could send its response
I20260929 17:58:13.054148 15043 ts_tablet_manager.cc:1905] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:58:13.057343 15043 ts_tablet_manager.cc:1918] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:58:13.057466 15043 log.cc:1196] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/wal/wals/4260a7533d0f4c16b822235a6b9edb22
I20260929 17:58:13.057811 15043 ts_tablet_manager.cc:1939] T 4260a7533d0f4c16b822235a6b9edb22 P 0dd99fb321114c759e2cf85a6fff9ee7: Deleting consensus metadata
W20260929 17:58:13.059031 14873 connection.cc:441] server connection from 127.0.0.1:55904 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55904 (request call id 1) could send its response
I20260929 17:58:13.061152 14995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:13.039091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58460 (local address 127.7.118.61:38403)
0929 17:58:13.039221 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:13.039227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:13.039245 (+ 18us) server_negotiation.cc:407] Connection header received
0929 17:58:13.039295 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:13.039299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:13.039353 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:13.039442 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:13.045721 (+ 6279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:13.046620 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:13.048003 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:13.048484 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:13.048638 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:13.057032 (+ 8394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:13.057036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:13.057039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:13.057070 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:13.059069 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:13.059640 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:13.059650 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:13.059652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:13.059757 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:13.060164 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:13.060167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:13.060169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:13.060389 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:13.060522 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:13.060831 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 17:58:13.060978 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 17:58:13.067025 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:13.053868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58468 (local address 127.7.118.61:38403)
0929 17:58:13.054005 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:13.054009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:13.055729 (+ 1720us) server_negotiation.cc:407] Connection header received
0929 17:58:13.055967 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:13.055971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:13.056035 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:13.056101 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:13.057011 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:13.057851 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:13.059431 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:13.059784 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:13.059866 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:13.062730 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:13.062734 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:13.062737 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:13.062764 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:13.064601 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:13.065524 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:13.065528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:13.065530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:13.065625 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:13.065976 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:13.065980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:13.065982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:13.066202 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:13.066355 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:13.066715 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 17:58:13.066846 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 17:58:13.070048 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:13.055395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58608 (local address 127.7.118.60:45305)
0929 17:58:13.055518 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:13.055524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:13.059854 (+ 4330us) server_negotiation.cc:407] Connection header received
0929 17:58:13.060073 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:13.060077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:13.060137 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:13.060207 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:13.061446 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:13.062392 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:13.063953 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:13.064279 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:13.064351 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:13.066223 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:13.066229 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:13.066233 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:13.066301 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:13.068135 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:13.068632 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:13.068636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:13.068638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:13.068743 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:13.069057 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:13.069060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:13.069062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:13.069282 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:13.069386 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:13.069749 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 17:58:13.069910 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 17:58:13.072376 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:13.039826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58604 (local address 127.7.118.60:45305)
0929 17:58:13.039951 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:13.039971 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:13.062362 (+ 22391us) server_negotiation.cc:407] Connection header received
0929 17:58:13.062410 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:13.062413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:13.062467 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:13.062546 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:13.063302 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:13.064036 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:13.065124 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:13.065510 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:13.065611 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:13.068072 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:13.068078 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:13.068082 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:13.068114 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:13.070557 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:13.071042 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:13.071046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:13.071048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:13.071126 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:13.071418 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:13.071420 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:13.071422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:13.071619 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:13.071759 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:13.072025 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 17:58:13.072152 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51,"spinlock_wait_cycles":2944}
I20260929 17:58:14.020946 15052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:14.020364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.7.118.1:33245)
0929 17:58:14.020710 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:14.020820 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":93,"threads_started":1}
W20260929 17:58:14.021441 14711 heartbeater.cc:644] Failed to heartbeat to 127.7.118.62:45353 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
I20260929 17:58:14.023090 15053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:14.022498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.7.118.2:54947)
0929 17:58:14.022884 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:14.022979 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":136,"threads_started":1}
W20260929 17:58:14.023399 14848 heartbeater.cc:644] Failed to heartbeat to 127.7.118.62:45353 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
I20260929 17:58:14.025645 15054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:14.025157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.7.118.3:41489)
0929 17:58:14.025434 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:14.025541 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":106,"threads_started":1}
W20260929 17:58:14.026104 14984 heartbeater.cc:644] Failed to heartbeat to 127.7.118.62:45353 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
I20260929 17:58:14.534888 15055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 796f18bdc8e94ff69ad6374576106f9f)
I20260929 17:58:14.535089 15055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: false } }
I20260929 17:58:14.535310 15056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 796f18bdc8e94ff69ad6374576106f9f)
I20260929 17:58:14.535521 15056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: false } }
I20260929 17:58:14.536154 15055 leader_election.cc:288] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 796f18bdc8e94ff69ad6374576106f9f (127.7.118.62:45353), 9df917bc96f642689729eb7f4803dffa (127.7.118.61:38403)
I20260929 17:58:14.536585 15057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:14.535851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.0.0.1:41142)
0929 17:58:14.536379 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:14.536466 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":68,"threads_started":1}
W20260929 17:58:14.536782 14517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
I20260929 17:58:14.537472 15056 leader_election.cc:288] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 796f18bdc8e94ff69ad6374576106f9f (127.7.118.62:45353), 9d44b2b97944484f88c630984887cf55 (127.7.118.60:45305)
I20260929 17:58:14.537844 15062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:14.536743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.0.0.1:41152)
0929 17:58:14.537679 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:14.537752 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":869,"thread_start_us":64,"threads_started":1}
I20260929 17:58:14.537972 15058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:14.537776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.0.0.1:41164)
0929 17:58:14.537856 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:14.537913 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 17:58:14.538003 14334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
W20260929 17:58:14.538106 14517 leader_election.cc:334] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 796f18bdc8e94ff69ad6374576106f9f (127.7.118.62:45353): Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
I20260929 17:58:14.539009 15062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:14.538780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.0.0.1:41170)
0929 17:58:14.538891 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:14.538943 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 17:58:14.539229 14334 leader_election.cc:334] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 796f18bdc8e94ff69ad6374576106f9f (127.7.118.62:45353): Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
I20260929 17:58:14.541653 15059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:14.537169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:45305 (local address 127.0.0.1:58612)
0929 17:58:14.538047 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:14.538062 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:14.538179 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:14.538505 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:14.538508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:14.538529 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:14.538629 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:14.538637 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:14.539576 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:14.539581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:14.540422 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:14.540436 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:14.541117 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:14.541121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:14.541270 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:14.541293 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 17:58:14.541391 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 17:58:14.541476 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":21,"thread_start_us":70,"threads_started":1}
I20260929 17:58:14.541657 15063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:14.537500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58612 (local address 127.7.118.60:45305)
0929 17:58:14.537732 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:14.537741 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:14.538298 (+ 557us) server_negotiation.cc:407] Connection header received
0929 17:58:14.538343 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:14.538346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:14.538402 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:14.538487 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:14.538734 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:14.539465 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:14.540553 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:14.540998 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:14.541114 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:14.541232 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:14.541406 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 17:58:14.541530 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":175,"thread_start_us":93,"threads_started":1}
I20260929 17:58:14.541675 15057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:14.536648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:38403 (local address 127.0.0.1:58480)
0929 17:58:14.537211 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:14.537228 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:14.537334 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:14.537863 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:14.537868 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:14.537979 (+ 111us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:14.538328 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:14.538336 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:14.539458 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:14.539468 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:14.540366 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:14.540375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:14.541119 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:14.541123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:14.541277 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:14.541301 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 17:58:14.541410 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 17:58:14.541523 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":28,"thread_start_us":62,"threads_started":1}
I20260929 17:58:14.541675 15061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:14.536934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58480 (local address 127.7.118.61:38403)
0929 17:58:14.537358 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:14.537365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:14.537384 (+ 19us) server_negotiation.cc:407] Connection header received
0929 17:58:14.537436 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:14.537440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:14.537487 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:14.537735 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:14.538462 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:14.539319 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:14.540498 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:14.540990 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:14.541187 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:14.541255 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:14.541429 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 17:58:14.541518 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":355,"thread_start_us":57,"threads_started":1}
I20260929 17:58:14.542550 14358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d44b2b97944484f88c630984887cf55" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9df917bc96f642689729eb7f4803dffa" is_pre_election: true
I20260929 17:58:14.542594 14541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9df917bc96f642689729eb7f4803dffa" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9d44b2b97944484f88c630984887cf55" is_pre_election: true
I20260929 17:58:14.542739 14358 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d44b2b97944484f88c630984887cf55 in term 1.
I20260929 17:58:14.542795 14541 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9df917bc96f642689729eb7f4803dffa in term 1.
I20260929 17:58:14.543112 14518 leader_election.cc:302] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d44b2b97944484f88c630984887cf55, 9df917bc96f642689729eb7f4803dffa; no voters: 796f18bdc8e94ff69ad6374576106f9f
I20260929 17:58:14.543156 14333 leader_election.cc:302] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d44b2b97944484f88c630984887cf55, 9df917bc96f642689729eb7f4803dffa; no voters: 796f18bdc8e94ff69ad6374576106f9f
I20260929 17:58:14.543362 15056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 17:58:14.543365 15055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 17:58:14.543432 15055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 796f18bdc8e94ff69ad6374576106f9f)
I20260929 17:58:14.543435 15056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 796f18bdc8e94ff69ad6374576106f9f)
I20260929 17:58:14.543478 15055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 1 FOLLOWER]: Advancing to term 2
I20260929 17:58:14.543493 15056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 1 FOLLOWER]: Advancing to term 2
I20260929 17:58:14.544524 15055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: false } }
I20260929 17:58:14.544559 15056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: false } }
I20260929 17:58:14.544934 15055 leader_election.cc:288] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [CANDIDATE]: Term 2 election: Requested vote from peers 796f18bdc8e94ff69ad6374576106f9f (127.7.118.62:45353), 9df917bc96f642689729eb7f4803dffa (127.7.118.61:38403)
I20260929 17:58:14.545116 15056 leader_election.cc:288] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [CANDIDATE]: Term 2 election: Requested vote from peers 796f18bdc8e94ff69ad6374576106f9f (127.7.118.62:45353), 9d44b2b97944484f88c630984887cf55 (127.7.118.60:45305)
I20260929 17:58:14.545419 15057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:14.545046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.0.0.1:41182)
0929 17:58:14.545257 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:14.545317 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260929 17:58:14.545410 15059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:14.545097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.0.0.1:41190)
0929 17:58:14.545300 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:14.545348 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260929 17:58:14.545424 14358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d44b2b97944484f88c630984887cf55" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9df917bc96f642689729eb7f4803dffa"
I20260929 17:58:14.545616 14358 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d44b2b97944484f88c630984887cf55 in current term 2: Already voted for candidate 9df917bc96f642689729eb7f4803dffa in this term.
I20260929 17:58:14.546059 14541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9df917bc96f642689729eb7f4803dffa" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9d44b2b97944484f88c630984887cf55"
I20260929 17:58:14.546236 14541 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9df917bc96f642689729eb7f4803dffa in current term 2: Already voted for candidate 9d44b2b97944484f88c630984887cf55 in this term.
I20260929 17:58:14.546416 15059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:14.546136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.0.0.1:41204)
0929 17:58:14.546279 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:14.546344 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 17:58:14.546442 15057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:14.545973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.0.0.1:41194)
0929 17:58:14.546320 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:14.546369 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253}
W20260929 17:58:14.546515 14334 leader_election.cc:334] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 796f18bdc8e94ff69ad6374576106f9f (127.7.118.62:45353): Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
I20260929 17:58:14.546599 14333 leader_election.cc:302] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9df917bc96f642689729eb7f4803dffa; no voters: 796f18bdc8e94ff69ad6374576106f9f, 9d44b2b97944484f88c630984887cf55
I20260929 17:58:14.546775 15056 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260929 17:58:14.546869 14517 leader_election.cc:334] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 796f18bdc8e94ff69ad6374576106f9f (127.7.118.62:45353): Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
I20260929 17:58:14.546948 14517 leader_election.cc:302] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d44b2b97944484f88c630984887cf55; no voters: 796f18bdc8e94ff69ad6374576106f9f, 9df917bc96f642689729eb7f4803dffa
I20260929 17:58:14.547087 15055 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 17:58:15.022666 15065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:15.022168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.7.118.1:44757)
0929 17:58:15.022471 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:15.022549 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":119,"threads_started":1}
I20260929 17:58:15.024519 15066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:15.024058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.7.118.2:34707)
0929 17:58:15.024349 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:15.024414 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":80,"threads_started":1}
I20260929 17:58:15.027107 15067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:15.026670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.7.118.3:33845)
0929 17:58:15.026922 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:15.026979 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":87,"threads_started":1}
I20260929 17:58:16.024056 15068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.023589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.7.118.1:54739)
0929 17:58:16.023848 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:16.023909 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":85,"threads_started":1}
I20260929 17:58:16.025854 15069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.025350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.7.118.2:56255)
0929 17:58:16.025662 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:16.025729 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":74,"threads_started":1}
I20260929 17:58:16.028172 15070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.027773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.7.118.3:41465)
0929 17:58:16.028034 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:16.028089 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":120,"threads_started":1}
I20260929 17:58:16.141130 15071 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:58:16.141281 15071 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: false } }
I20260929 17:58:16.141750 15071 leader_election.cc:288] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 796f18bdc8e94ff69ad6374576106f9f (127.7.118.62:45353), 9d44b2b97944484f88c630984887cf55 (127.7.118.60:45305)
I20260929 17:58:16.142379 14541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9df917bc96f642689729eb7f4803dffa" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9d44b2b97944484f88c630984887cf55" is_pre_election: true
I20260929 17:58:16.142588 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.142026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.0.0.1:41472)
0929 17:58:16.142390 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:16.142471 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":138,"threads_started":1}
I20260929 17:58:16.142571 14541 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9df917bc96f642689729eb7f4803dffa in term 2.
I20260929 17:58:16.142899 14333 leader_election.cc:302] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d44b2b97944484f88c630984887cf55, 9df917bc96f642689729eb7f4803dffa; no voters:
I20260929 17:58:16.143168 15071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 17:58:16.143254 15071 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:58:16.143313 15071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 2 FOLLOWER]: Advancing to term 3
I20260929 17:58:16.143751 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.143402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.0.0.1:41482)
0929 17:58:16.143600 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:16.143664 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260929 17:58:16.143893 14334 leader_election.cc:334] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 796f18bdc8e94ff69ad6374576106f9f (127.7.118.62:45353): Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
I20260929 17:58:16.144696 15071 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: false } }
I20260929 17:58:16.145128 15071 leader_election.cc:288] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [CANDIDATE]: Term 3 election: Requested vote from peers 796f18bdc8e94ff69ad6374576106f9f (127.7.118.62:45353), 9d44b2b97944484f88c630984887cf55 (127.7.118.60:45305)
I20260929 17:58:16.145562 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.145306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.0.0.1:41496)
0929 17:58:16.145426 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:16.145489 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 17:58:16.145571 14541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9df917bc96f642689729eb7f4803dffa" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9d44b2b97944484f88c630984887cf55"
I20260929 17:58:16.145759 14541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 2 FOLLOWER]: Advancing to term 3
I20260929 17:58:16.146322 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.146048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.0.0.1:41506)
0929 17:58:16.146157 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:16.146220 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 17:58:16.146440 14334 leader_election.cc:334] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 796f18bdc8e94ff69ad6374576106f9f (127.7.118.62:45353): Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
I20260929 17:58:16.147037 14541 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9df917bc96f642689729eb7f4803dffa in term 3.
I20260929 17:58:16.147418 14333 leader_election.cc:302] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d44b2b97944484f88c630984887cf55, 9df917bc96f642689729eb7f4803dffa; no voters: 796f18bdc8e94ff69ad6374576106f9f
I20260929 17:58:16.147601 15071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 3 FOLLOWER]: Leader election won for term 3
I20260929 17:58:16.147843 15071 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [term 3 LEADER]: Becoming Leader. State: Replica: 9df917bc96f642689729eb7f4803dffa, State: Running, Role: LEADER
I20260929 17:58:16.148160 15071 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: false } }
I20260929 17:58:16.149304 15074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9df917bc96f642689729eb7f4803dffa. Latest consensus state: current_term: 3 leader_uuid: "9df917bc96f642689729eb7f4803dffa" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: false } } }
I20260929 17:58:16.149448 15074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:16.149808 15076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:58:16.152443 15076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:58:16.152940 15076 catalog_manager.cc:1259] Loaded cluster ID: dbce91949f94432f864ae3fbb8d5ca2e
I20260929 17:58:16.152987 15076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:58:16.153590 15076 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:58:16.154076 15076 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa: Loaded TSK: 0
I20260929 17:58:16.154872 15076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:58:16.216158 14348 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:58:16.219486 14348 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:58:16.223071 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.222794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.0.0.1:41526)
0929 17:58:16.222923 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:16.222989 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 17:58:16.223136 14541 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [term 3 FOLLOWER]: Refusing update from remote peer 9df917bc96f642689729eb7f4803dffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260929 17:58:16.223850 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.223609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.0.0.1:41534)
0929 17:58:16.223718 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:16.223778 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 17:58:16.223742 15074 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260929 17:58:16.224004 14334 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa -> Peer 796f18bdc8e94ff69ad6374576106f9f (127.7.118.62:45353): Couldn't send request to peer 796f18bdc8e94ff69ad6374576106f9f. Status: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 17:58:16.225515 15078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9df917bc96f642689729eb7f4803dffa. Latest consensus state: current_term: 3 leader_uuid: "9df917bc96f642689729eb7f4803dffa" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: false } } }
I20260929 17:58:16.225689 15078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:58:16.226796 15077 mvcc.cc:205] Tried to move back new op lower bound from 7334726435723710464 to 7334726435425218560. Current Snapshot: MvccSnapshot[applied={T|T < 7334726435723710464}]
I20260929 17:58:16.227017 15071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "9df917bc96f642689729eb7f4803dffa" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: false } } }
I20260929 17:58:16.227147 15071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:16.227329 15079 mvcc.cc:205] Tried to move back new op lower bound from 7334726435723710464 to 7334726435425218560. Current Snapshot: MvccSnapshot[applied={T|T < 7334726435723710464}]
I20260929 17:58:16.227413 15078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "9df917bc96f642689729eb7f4803dffa" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "796f18bdc8e94ff69ad6374576106f9f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45353 } } peers { permanent_uuid: "9df917bc96f642689729eb7f4803dffa" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 38403 } attrs { promote: false } } peers { permanent_uuid: "9d44b2b97944484f88c630984887cf55" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 45305 } attrs { promote: false } } }
I20260929 17:58:16.227532 15078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d44b2b97944484f88c630984887cf55 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:58:16.240577 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.235497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:37857 (local address 127.0.0.1:55924)
0929 17:58:16.235643 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:16.235662 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:16.235793 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:16.236637 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:16.236640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:16.236652 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:16.236720 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:16.236730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:16.238172 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:16.238175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:16.239152 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:16.239166 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:16.240143 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:16.240149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:16.240251 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:16.240274 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 17:58:16.240325 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 17:58:16.240401 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":42}
I20260929 17:58:16.240722 15085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.235862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34536 (local address 127.7.118.1:42087)
0929 17:58:16.236143 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:16.236147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:16.236160 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:58:16.236234 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:16.236237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:16.236290 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:16.236389 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:16.236839 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:16.237675 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:16.239436 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:16.240090 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:16.240199 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:16.240279 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:16.240490 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 17:58:16.240572 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":118,"threads_started":1}
I20260929 17:58:16.240901 15084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.235744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:42087 (local address 127.0.0.1:34536)
0929 17:58:16.236016 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:16.236030 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:16.236121 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:16.236531 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:16.236538 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:16.236561 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:16.236691 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:16.236698 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:16.237812 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:16.237815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:16.239285 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:16.239297 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:16.240228 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:16.240232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:16.240352 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:16.240377 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 17:58:16.240461 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 17:58:16.240544 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":27,"thread_start_us":137,"threads_started":1}
I20260929 17:58:16.240911 15087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.236391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:36521 (local address 127.0.0.1:33064)
0929 17:58:16.236663 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:16.236676 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:16.236764 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:16.237232 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:16.237235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:16.237251 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:16.237346 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:16.237352 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:16.238338 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:16.238342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:16.239544 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:16.239552 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:16.240482 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:16.240486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:16.240570 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:16.240593 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 17:58:16.240673 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 17:58:16.240749 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":148,"threads_started":1}
I20260929 17:58:16.241552 14644 tablet_service.cc:1467] Processing CreateTablet for tablet 6c92a37c9e8f40b3856e36875a6eec4f (DEFAULT_TABLE table=test-table [id=33a8443fbbf04b86a807e98b31e22783]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:58:16.241640 15086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.236078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55924 (local address 127.7.118.3:37857)
0929 17:58:16.236345 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:16.236353 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:16.236371 (+ 18us) server_negotiation.cc:407] Connection header received
0929 17:58:16.236428 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:16.236431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:16.236479 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:16.236836 (+ 357us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:16.236943 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:16.238071 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:16.239315 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:16.239990 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:16.241203 (+ 1213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:16.241272 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:16.241310 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 17:58:16.241385 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":79,"threads_started":1}
I20260929 17:58:16.241959 14644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c92a37c9e8f40b3856e36875a6eec4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:16.242172 14917 tablet_service.cc:1467] Processing CreateTablet for tablet 6c92a37c9e8f40b3856e36875a6eec4f (DEFAULT_TABLE table=test-table [id=33a8443fbbf04b86a807e98b31e22783]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:58:16.242511 14917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c92a37c9e8f40b3856e36875a6eec4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:16.242585 15088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.236729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33064 (local address 127.7.118.2:36521)
0929 17:58:16.237025 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:16.237029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:16.237040 (+ 11us) server_negotiation.cc:407] Connection header received
0929 17:58:16.237083 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:16.237085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:16.237119 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:16.237212 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:16.237458 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:16.238208 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:16.239682 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:16.240233 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:16.240349 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:16.240421 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:16.242383 (+ 1962us) server_negotiation.cc:299] Negotiation successful
0929 17:58:16.242472 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":66,"threads_started":1}
I20260929 17:58:16.243150 14781 tablet_service.cc:1467] Processing CreateTablet for tablet 6c92a37c9e8f40b3856e36875a6eec4f (DEFAULT_TABLE table=test-table [id=33a8443fbbf04b86a807e98b31e22783]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:58:16.243373 14781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c92a37c9e8f40b3856e36875a6eec4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:16.245661 15091 tablet_bootstrap.cc:492] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52: Bootstrap starting.
I20260929 17:58:16.245841 15090 tablet_bootstrap.cc:492] T 6c92a37c9e8f40b3856e36875a6eec4f P 0dd99fb321114c759e2cf85a6fff9ee7: Bootstrap starting.
I20260929 17:58:16.246462 15089 tablet_bootstrap.cc:492] T 6c92a37c9e8f40b3856e36875a6eec4f P 38ba528a27114efbae960672fb79404b: Bootstrap starting.
I20260929 17:58:16.246801 15091 tablet_bootstrap.cc:654] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:16.246829 15090 tablet_bootstrap.cc:654] T 6c92a37c9e8f40b3856e36875a6eec4f P 0dd99fb321114c759e2cf85a6fff9ee7: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:16.247440 15089 tablet_bootstrap.cc:654] T 6c92a37c9e8f40b3856e36875a6eec4f P 38ba528a27114efbae960672fb79404b: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:16.248528 15091 tablet_bootstrap.cc:492] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52: No bootstrap required, opened a new log
I20260929 17:58:16.248528 15090 tablet_bootstrap.cc:492] T 6c92a37c9e8f40b3856e36875a6eec4f P 0dd99fb321114c759e2cf85a6fff9ee7: No bootstrap required, opened a new log
I20260929 17:58:16.248637 15091 ts_tablet_manager.cc:1397] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 17:58:16.248649 15090 ts_tablet_manager.cc:1397] T 6c92a37c9e8f40b3856e36875a6eec4f P 0dd99fb321114c759e2cf85a6fff9ee7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 17:58:16.249167 15091 raft_consensus.cc:348] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d13d7d66af745589d65193cd03d8f52" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 36521 } } peers { permanent_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37857 } } peers { permanent_uuid: "38ba528a27114efbae960672fb79404b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42087 } }
I20260929 17:58:16.249167 15090 raft_consensus.cc:348] T 6c92a37c9e8f40b3856e36875a6eec4f P 0dd99fb321114c759e2cf85a6fff9ee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d13d7d66af745589d65193cd03d8f52" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 36521 } } peers { permanent_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37857 } } peers { permanent_uuid: "38ba528a27114efbae960672fb79404b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42087 } }
I20260929 17:58:16.249308 15090 raft_consensus.cc:374] T 6c92a37c9e8f40b3856e36875a6eec4f P 0dd99fb321114c759e2cf85a6fff9ee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:16.249308 15091 raft_consensus.cc:374] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:16.249348 15090 raft_consensus.cc:729] T 6c92a37c9e8f40b3856e36875a6eec4f P 0dd99fb321114c759e2cf85a6fff9ee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dd99fb321114c759e2cf85a6fff9ee7, State: Initialized, Role: FOLLOWER
I20260929 17:58:16.249348 15091 raft_consensus.cc:729] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d13d7d66af745589d65193cd03d8f52, State: Initialized, Role: FOLLOWER
I20260929 17:58:16.249517 15090 consensus_queue.cc:260] T 6c92a37c9e8f40b3856e36875a6eec4f P 0dd99fb321114c759e2cf85a6fff9ee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d13d7d66af745589d65193cd03d8f52" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 36521 } } peers { permanent_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37857 } } peers { permanent_uuid: "38ba528a27114efbae960672fb79404b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42087 } }
I20260929 17:58:16.249562 15091 consensus_queue.cc:260] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d13d7d66af745589d65193cd03d8f52" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 36521 } } peers { permanent_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37857 } } peers { permanent_uuid: "38ba528a27114efbae960672fb79404b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42087 } }
I20260929 17:58:16.249960 15090 ts_tablet_manager.cc:1428] T 6c92a37c9e8f40b3856e36875a6eec4f P 0dd99fb321114c759e2cf85a6fff9ee7: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 17:58:16.250425 15091 ts_tablet_manager.cc:1428] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 17:58:16.251235 15069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.250854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.7.118.2:34577)
0929 17:58:16.251046 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:16.251108 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 17:58:16.251523 15070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.250983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.7.118.3:45149)
0929 17:58:16.251380 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:16.251440 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316}
I20260929 17:58:16.251959 15069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.251745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.7.118.2:53751)
0929 17:58:16.251875 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:16.251912 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 17:58:16.254065 14849 heartbeater.cc:503] Master 127.7.118.61:38403 was elected leader, sending a full tablet report...
I20260929 17:58:16.255702 15089 tablet_bootstrap.cc:492] T 6c92a37c9e8f40b3856e36875a6eec4f P 38ba528a27114efbae960672fb79404b: No bootstrap required, opened a new log
I20260929 17:58:16.255828 15089 ts_tablet_manager.cc:1397] T 6c92a37c9e8f40b3856e36875a6eec4f P 38ba528a27114efbae960672fb79404b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260929 17:58:16.256256 15089 raft_consensus.cc:348] T 6c92a37c9e8f40b3856e36875a6eec4f P 38ba528a27114efbae960672fb79404b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d13d7d66af745589d65193cd03d8f52" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 36521 } } peers { permanent_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37857 } } peers { permanent_uuid: "38ba528a27114efbae960672fb79404b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42087 } }
I20260929 17:58:16.256374 15089 raft_consensus.cc:374] T 6c92a37c9e8f40b3856e36875a6eec4f P 38ba528a27114efbae960672fb79404b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:16.256418 15089 raft_consensus.cc:729] T 6c92a37c9e8f40b3856e36875a6eec4f P 38ba528a27114efbae960672fb79404b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ba528a27114efbae960672fb79404b, State: Initialized, Role: FOLLOWER
I20260929 17:58:16.256563 15089 consensus_queue.cc:260] T 6c92a37c9e8f40b3856e36875a6eec4f P 38ba528a27114efbae960672fb79404b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d13d7d66af745589d65193cd03d8f52" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 36521 } } peers { permanent_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37857 } } peers { permanent_uuid: "38ba528a27114efbae960672fb79404b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42087 } }
I20260929 17:58:16.256937 15089 ts_tablet_manager.cc:1428] T 6c92a37c9e8f40b3856e36875a6eec4f P 38ba528a27114efbae960672fb79404b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 17:58:16.257678 15068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.257378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.7.118.1:50583)
0929 17:58:16.257494 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:16.257577 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 17:58:16.258890 14712 heartbeater.cc:503] Master 127.7.118.61:38403 was elected leader, sending a full tablet report...
I20260929 17:58:16.259172 15068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.258900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.7.118.1:57981)
0929 17:58:16.259055 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:16.259112 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 17:58:16.307790 15095 raft_consensus.cc:482] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:58:16.307945 15095 raft_consensus.cc:504] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d13d7d66af745589d65193cd03d8f52" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 36521 } } peers { permanent_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37857 } } peers { permanent_uuid: "38ba528a27114efbae960672fb79404b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42087 } }
I20260929 17:58:16.309158 15095 leader_election.cc:288] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dd99fb321114c759e2cf85a6fff9ee7 (127.7.118.3:37857), 38ba528a27114efbae960672fb79404b (127.7.118.1:42087)
I20260929 17:58:16.314893 15085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.309666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:60073 (local address 127.7.118.1:42087)
0929 17:58:16.309818 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:16.309823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:16.309998 (+ 175us) server_negotiation.cc:407] Connection header received
0929 17:58:16.310158 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:16.310164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:16.310216 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:16.310351 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:16.310620 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:16.311400 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:16.313451 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:16.314147 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:16.314268 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:16.314322 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:16.314679 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 17:58:16.314752 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 17:58:16.314893 15069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.309365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:37857 (local address 127.7.118.2:32791)
0929 17:58:16.309694 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:16.309729 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:16.309853 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:16.310184 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:16.310186 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:16.310200 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:16.310401 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:16.310409 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:16.311540 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:16.311551 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:16.313256 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:16.313274 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:16.314391 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:16.314395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:16.314557 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:16.314582 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 17:58:16.314657 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 17:58:16.314739 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":147}
I20260929 17:58:16.314905 15098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.309517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:42087 (local address 127.7.118.2:60073)
0929 17:58:16.309895 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:16.309909 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:16.310054 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:16.310373 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:16.310377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:16.310392 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:16.310499 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:16.310509 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:16.311513 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:16.311518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:16.313256 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:16.313274 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:16.314310 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:16.314314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:16.314452 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:16.314485 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 17:58:16.314608 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 17:58:16.314710 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":135,"thread_start_us":134,"threads_started":1}
I20260929 17:58:16.315379 15086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.309426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:32791 (local address 127.7.118.3:37857)
0929 17:58:16.309573 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:16.309577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:16.309940 (+ 363us) server_negotiation.cc:407] Connection header received
0929 17:58:16.310028 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:16.310031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:16.310086 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:16.310179 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:16.310528 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:16.311338 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:16.313451 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:16.314089 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:16.314225 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:16.314323 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:16.315156 (+ 833us) server_negotiation.cc:299] Negotiation successful
0929 17:58:16.315256 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 17:58:16.315444 14664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c92a37c9e8f40b3856e36875a6eec4f" candidate_uuid: "6d13d7d66af745589d65193cd03d8f52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ba528a27114efbae960672fb79404b" is_pre_election: true
I20260929 17:58:16.315647 14664 raft_consensus.cc:2454] T 6c92a37c9e8f40b3856e36875a6eec4f P 38ba528a27114efbae960672fb79404b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d13d7d66af745589d65193cd03d8f52 in term 0.
I20260929 17:58:16.315827 14937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c92a37c9e8f40b3856e36875a6eec4f" candidate_uuid: "6d13d7d66af745589d65193cd03d8f52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" is_pre_election: true
I20260929 17:58:16.315959 14937 raft_consensus.cc:2454] T 6c92a37c9e8f40b3856e36875a6eec4f P 0dd99fb321114c759e2cf85a6fff9ee7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d13d7d66af745589d65193cd03d8f52 in term 0.
I20260929 17:58:16.316062 14737 leader_election.cc:302] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ba528a27114efbae960672fb79404b, 6d13d7d66af745589d65193cd03d8f52; no voters:
I20260929 17:58:16.316380 15095 raft_consensus.cc:2781] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:58:16.316442 15095 raft_consensus.cc:482] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:58:16.316480 15095 raft_consensus.cc:3037] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:16.317554 15095 raft_consensus.cc:504] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d13d7d66af745589d65193cd03d8f52" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 36521 } } peers { permanent_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37857 } } peers { permanent_uuid: "38ba528a27114efbae960672fb79404b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42087 } }
I20260929 17:58:16.317916 15095 leader_election.cc:288] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52 [CANDIDATE]: Term 1 election: Requested vote from peers 0dd99fb321114c759e2cf85a6fff9ee7 (127.7.118.3:37857), 38ba528a27114efbae960672fb79404b (127.7.118.1:42087)
I20260929 17:58:16.318361 14664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c92a37c9e8f40b3856e36875a6eec4f" candidate_uuid: "6d13d7d66af745589d65193cd03d8f52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ba528a27114efbae960672fb79404b"
I20260929 17:58:16.318470 14664 raft_consensus.cc:3037] T 6c92a37c9e8f40b3856e36875a6eec4f P 38ba528a27114efbae960672fb79404b [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:16.318436 14937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c92a37c9e8f40b3856e36875a6eec4f" candidate_uuid: "6d13d7d66af745589d65193cd03d8f52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dd99fb321114c759e2cf85a6fff9ee7"
I20260929 17:58:16.318581 14937 raft_consensus.cc:3037] T 6c92a37c9e8f40b3856e36875a6eec4f P 0dd99fb321114c759e2cf85a6fff9ee7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:16.319450 14664 raft_consensus.cc:2454] T 6c92a37c9e8f40b3856e36875a6eec4f P 38ba528a27114efbae960672fb79404b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d13d7d66af745589d65193cd03d8f52 in term 1.
I20260929 17:58:16.319705 14937 raft_consensus.cc:2454] T 6c92a37c9e8f40b3856e36875a6eec4f P 0dd99fb321114c759e2cf85a6fff9ee7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d13d7d66af745589d65193cd03d8f52 in term 1.
I20260929 17:58:16.319778 14737 leader_election.cc:302] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ba528a27114efbae960672fb79404b, 6d13d7d66af745589d65193cd03d8f52; no voters:
I20260929 17:58:16.319983 15095 raft_consensus.cc:2781] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:58:16.320255 15095 raft_consensus.cc:686] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52 [term 1 LEADER]: Becoming Leader. State: Replica: 6d13d7d66af745589d65193cd03d8f52, State: Running, Role: LEADER
I20260929 17:58:16.320549 15095 consensus_queue.cc:237] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d13d7d66af745589d65193cd03d8f52" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 36521 } } peers { permanent_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37857 } } peers { permanent_uuid: "38ba528a27114efbae960672fb79404b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42087 } }
I20260929 17:58:16.321107 15098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.320798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.7.118.2:35885)
0929 17:58:16.320935 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:16.321036 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 17:58:16.323534 14347 catalog_manager.cc:5495] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d13d7d66af745589d65193cd03d8f52 (127.7.118.2). New cstate: current_term: 1 leader_uuid: "6d13d7d66af745589d65193cd03d8f52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d13d7d66af745589d65193cd03d8f52" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 36521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38ba528a27114efbae960672fb79404b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42087 } health_report { overall_health: UNKNOWN } } }
I20260929 17:58:16.343410 15088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.339146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33076 (local address 127.7.118.2:36521)
0929 17:58:16.339277 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:16.339281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:16.339294 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:58:16.339482 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:16.339485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:16.339547 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:16.339640 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:58:16.339925 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:16.340476 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:16.341490 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:16.341739 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:16.341829 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:16.342161 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:58:16.342748 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:58:16.342849 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:16.343116 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 17:58:16.343227 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 17:58:16.346485 14664 raft_consensus.cc:1261] T 6c92a37c9e8f40b3856e36875a6eec4f P 38ba528a27114efbae960672fb79404b [term 1 FOLLOWER]: Refusing update from remote peer 6d13d7d66af745589d65193cd03d8f52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:58:16.346740 14937 raft_consensus.cc:1261] T 6c92a37c9e8f40b3856e36875a6eec4f P 0dd99fb321114c759e2cf85a6fff9ee7 [term 1 FOLLOWER]: Refusing update from remote peer 6d13d7d66af745589d65193cd03d8f52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:58:16.347172 15099 consensus_queue.cc:1035] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38ba528a27114efbae960672fb79404b" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:58:16.347389 15095 consensus_queue.cc:1035] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dd99fb321114c759e2cf85a6fff9ee7" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:58:16.350551 15068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.350093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.7.118.1:44609)
0929 17:58:16.350370 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:16.350452 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
I20260929 17:58:16.350719 14985 heartbeater.cc:503] Master 127.7.118.61:38403 was elected leader, sending a full tablet report...
I20260929 17:58:16.351495 15070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.351175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.7.118.3:53035)
0929 17:58:16.351329 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:16.351404 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 17:58:16.354848 15098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.354542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.7.118.2:44061)
0929 17:58:16.354676 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:16.354751 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 17:58:16.355195 15103 mvcc.cc:205] Tried to move back new op lower bound from 7334726436229656576 to 7334726436131958784. Current Snapshot: MvccSnapshot[applied={T|T < 7334726436229656576}]
I20260929 17:58:16.355379 15068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.355114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.7.118.1:35255)
0929 17:58:16.355263 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:16.355315 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 17:58:16.356614 15070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.356296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45353 (local address 127.7.118.3:35405)
0929 17:58:16.356469 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:16.356535 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 17:58:16.357558 15105 mvcc.cc:205] Tried to move back new op lower bound from 7334726436229656576 to 7334726436131958784. Current Snapshot: MvccSnapshot[applied={T|T < 7334726436229656576}]
I20260929 17:58:16.367053 14347 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:58:16.367238 14347 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:58:16.372423 14347 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9df917bc96f642689729eb7f4803dffa: Sending DeleteTablet for 3 replicas of tablet 6c92a37c9e8f40b3856e36875a6eec4f
I20260929 17:58:16.373301 14781 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c92a37c9e8f40b3856e36875a6eec4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:58:16 UTC) from {username='oryx', principal='oryx/127.7.118.61@KRBTEST.COM'} at 127.0.0.1:33064
I20260929 17:58:16.373437 14644 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c92a37c9e8f40b3856e36875a6eec4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:58:16 UTC) from {username='oryx', principal='oryx/127.7.118.61@KRBTEST.COM'} at 127.0.0.1:34536
I20260929 17:58:16.373723 15112 tablet_replica.cc:330] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52: stopping tablet replica
I20260929 17:58:16.373862 15112 raft_consensus.cc:2229] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:58:16.373973 14917 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c92a37c9e8f40b3856e36875a6eec4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:58:16 UTC) from {username='oryx', principal='oryx/127.7.118.61@KRBTEST.COM'} at 127.0.0.1:55924
I20260929 17:58:16.374286 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 14492
I20260929 17:58:16.374280 15112 raft_consensus.cc:2258] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:58:16.374346 15114 tablet_replica.cc:330] T 6c92a37c9e8f40b3856e36875a6eec4f P 0dd99fb321114c759e2cf85a6fff9ee7: stopping tablet replica
I20260929 17:58:16.375586 15114 raft_consensus.cc:2229] T 6c92a37c9e8f40b3856e36875a6eec4f P 0dd99fb321114c759e2cf85a6fff9ee7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:58:16.375753 15114 raft_consensus.cc:2258] T 6c92a37c9e8f40b3856e36875a6eec4f P 0dd99fb321114c759e2cf85a6fff9ee7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:58:16.376708 15114 ts_tablet_manager.cc:1905] T 6c92a37c9e8f40b3856e36875a6eec4f P 0dd99fb321114c759e2cf85a6fff9ee7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:58:16.376982 15112 ts_tablet_manager.cc:1905] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:58:16.380293 15112 ts_tablet_manager.cc:1918] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:58:16.380349 15114 ts_tablet_manager.cc:1918] T 6c92a37c9e8f40b3856e36875a6eec4f P 0dd99fb321114c759e2cf85a6fff9ee7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:58:16.380402 15112 log.cc:1196] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-1/wal/wals/6c92a37c9e8f40b3856e36875a6eec4f
I20260929 17:58:16.380439 15114 log.cc:1196] T 6c92a37c9e8f40b3856e36875a6eec4f P 0dd99fb321114c759e2cf85a6fff9ee7: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704657566865-7640-0/minicluster-data/ts-2/wal/wals/6c92a37c9e8f40b3856e36875a6eec4f
I20260929 17:58:16.380693 15112 ts_tablet_manager.cc:1939] T 6c92a37c9e8f40b3856e36875a6eec4f P 6d13d7d66af745589d65193cd03d8f52: Deleting consensus metadata
I20260929 17:58:16.380729 15114 ts_tablet_manager.cc:1939] T 6c92a37c9e8f40b3856e36875a6eec4f P 0dd99fb321114c759e2cf85a6fff9ee7: Deleting consensus metadata
I20260929 17:58:16.381580 14333 catalog_manager.cc:4841] TS 6d13d7d66af745589d65193cd03d8f52 (127.7.118.2:36521): tablet 6c92a37c9e8f40b3856e36875a6eec4f (table test-table [id=33a8443fbbf04b86a807e98b31e22783]) successfully deleted
I20260929 17:58:16.381754 14334 catalog_manager.cc:4841] TS 0dd99fb321114c759e2cf85a6fff9ee7 (127.7.118.3:37857): tablet 6c92a37c9e8f40b3856e36875a6eec4f (table test-table [id=33a8443fbbf04b86a807e98b31e22783]) successfully deleted
I20260929 17:58:16.383457 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 14718
I20260929 17:58:16.383880 15084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:16.383503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:42087 (local address 127.0.0.1:52664)
0929 17:58:16.383703 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:16.383771 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.1:42087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260929 17:58:16.385273 14335 catalog_manager.cc:4584] TS 38ba528a27114efbae960672fb79404b (127.7.118.1:42087): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c92a37c9e8f40b3856e36875a6eec4f: Network error: Client connection negotiation failed: client connection to 127.7.118.1:42087: connect: Connection refused (error 111)
I20260929 17:58:16.389503 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 14854
I20260929 17:58:16.397018 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 14211
I20260929 17:58:16.403307 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 14312
2026-09-29T17:58:16Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7398 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:58:16 dist-test-slave-r7ph krb5kdc[15117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:58:16 dist-test-slave-r7ph krb5kdc[15117](info): set up 2 sockets
Sep 29 17:58:16 dist-test-slave-r7ph krb5kdc[15117](info): commencing operation
krb5kdc: starting...
W20260929 17:58:17.482993 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 17:58:17 dist-test-slave-r7ph krb5kdc[15117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:58:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:58:17Z Disabled control of system clock
WARNING: no policy specified for oryx/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.62.keytab.
Entry for principal oryx/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.62.keytab.
I20260929 17:58:17.630885 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:46821
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.62.keytab
--principal=oryx/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:46821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:17.755277 15130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:17.755635 15130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:17.755679 15130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:17.760015 15130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:58:17.760100 15130 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 17:58:17.760118 15130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:17.760136 15130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:17.760154 15130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:58:17.760169 15130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:58:17.765790 15130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:46821
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.62.keytab
--principal=oryx/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:46821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:17.766950 15130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:17.768179 15130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:17.777801 15138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:17.777801 15135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:17.777876 15130 server_base.cc:1034] running on GCE node
W20260929 17:58:17.777880 15136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:17.778856 15130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:17.780268 15130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:17.781455 15130 hybrid_clock.cc:648] HybridClock initialized: now 1790704697781438 us; error 40 us; skew 500 ppm
Sep 29 17:58:17 dist-test-slave-r7ph krb5kdc[15117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704697, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:17.784758 15130 init.cc:377] Logged in from keytab as oryx/127.7.118.62@KRBTEST.COM (short username oryx)
I20260929 17:58:17.785787 15130 webserver.cc:460] Webserver started at http://127.7.118.62:42091/ using document root <none> and password file <none>
I20260929 17:58:17.786443 15130 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:17.786520 15130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:17.786702 15130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:17.789255 15130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "0fbcb6f395034e0c8288ed78a863662c"
format_stamp: "Formatted at 2026-09-29 17:58:17 on dist-test-slave-r7ph"
server_key: "26134fa020bbb187b2937bbc812555a4"
server_key_iv: "c08c35996257e128059602b1fab4a953"
server_key_version: "encryptionkey@0"
I20260929 17:58:17.789837 15130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "0fbcb6f395034e0c8288ed78a863662c"
format_stamp: "Formatted at 2026-09-29 17:58:17 on dist-test-slave-r7ph"
server_key: "26134fa020bbb187b2937bbc812555a4"
server_key_iv: "c08c35996257e128059602b1fab4a953"
server_key_version: "encryptionkey@0"
I20260929 17:58:17.793938 15130 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 17:58:17.796936 15145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:17.798743 15130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:58:17.798898 15130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "0fbcb6f395034e0c8288ed78a863662c"
format_stamp: "Formatted at 2026-09-29 17:58:17 on dist-test-slave-r7ph"
server_key: "26134fa020bbb187b2937bbc812555a4"
server_key_iv: "c08c35996257e128059602b1fab4a953"
server_key_version: "encryptionkey@0"
I20260929 17:58:17.799014 15130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:17.811216 15130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:17.815258 15130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:17.815446 15130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:17.824910 15130 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:46821
I20260929 17:58:17.825026 15207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:46821 every 8 connection(s)
I20260929 17:58:17.826151 15130 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 17:58:17.828538 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 15130
I20260929 17:58:17.828703 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 17:58:17.829039 7640 external_mini_cluster.cc:1428] Setting key 0c39658a0a919bad98b95196ab0f7f8e
I20260929 17:58:17.829926 15208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:17.837785 15208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fbcb6f395034e0c8288ed78a863662c: Bootstrap starting.
Sep 29 17:58:17 dist-test-slave-r7ph krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM
I20260929 17:58:17.840885 15208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fbcb6f395034e0c8288ed78a863662c: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:17.842038 15208 log.cc:824] T 00000000000000000000000000000000 P 0fbcb6f395034e0c8288ed78a863662c: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:17.844547 15208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fbcb6f395034e0c8288ed78a863662c: No bootstrap required, opened a new log
I20260929 17:58:17.846097 15211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:17.830823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55328 (local address 127.7.118.62:46821)
0929 17:58:17.831523 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:17.831550 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:17.831598 (+ 48us) server_negotiation.cc:407] Connection header received
0929 17:58:17.832310 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:17.832330 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:17.832715 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:17.833057 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:17.834157 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:17.835324 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:17.836652 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:17.837293 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:17.837458 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:17.840769 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:17.840776 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:17.840792 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:17.840829 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:17.843008 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:17.843532 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:17.843538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:17.843546 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:17.843675 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:17.843991 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:17.843994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:17.843996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:17.844486 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:17.844696 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:17.844996 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 17:58:17.845311 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":182,"threads_started":1}
I20260929 17:58:17.848315 15208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fbcb6f395034e0c8288ed78a863662c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fbcb6f395034e0c8288ed78a863662c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 46821 } }
I20260929 17:58:17.848677 15208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fbcb6f395034e0c8288ed78a863662c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:17.848747 15208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fbcb6f395034e0c8288ed78a863662c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbcb6f395034e0c8288ed78a863662c, State: Initialized, Role: FOLLOWER
I20260929 17:58:17.849221 15208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fbcb6f395034e0c8288ed78a863662c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fbcb6f395034e0c8288ed78a863662c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 46821 } }
I20260929 17:58:17.849388 15208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fbcb6f395034e0c8288ed78a863662c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:58:17.849474 15208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fbcb6f395034e0c8288ed78a863662c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:58:17.849597 15208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fbcb6f395034e0c8288ed78a863662c [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:17.850821 15208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fbcb6f395034e0c8288ed78a863662c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fbcb6f395034e0c8288ed78a863662c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 46821 } }
I20260929 17:58:17.851441 15208 leader_election.cc:302] T 00000000000000000000000000000000 P 0fbcb6f395034e0c8288ed78a863662c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fbcb6f395034e0c8288ed78a863662c; no voters:
I20260929 17:58:17.851905 15208 leader_election.cc:288] T 00000000000000000000000000000000 P 0fbcb6f395034e0c8288ed78a863662c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:58:17.852074 15213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fbcb6f395034e0c8288ed78a863662c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:58:17.852341 15213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fbcb6f395034e0c8288ed78a863662c [term 1 LEADER]: Becoming Leader. State: Replica: 0fbcb6f395034e0c8288ed78a863662c, State: Running, Role: LEADER
I20260929 17:58:17.852725 15213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fbcb6f395034e0c8288ed78a863662c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fbcb6f395034e0c8288ed78a863662c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 46821 } }
I20260929 17:58:17.853402 15208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fbcb6f395034e0c8288ed78a863662c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:58:17.854753 15214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fbcb6f395034e0c8288ed78a863662c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fbcb6f395034e0c8288ed78a863662c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fbcb6f395034e0c8288ed78a863662c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 46821 } } }
I20260929 17:58:17.854833 15215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fbcb6f395034e0c8288ed78a863662c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fbcb6f395034e0c8288ed78a863662c. Latest consensus state: current_term: 1 leader_uuid: "0fbcb6f395034e0c8288ed78a863662c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fbcb6f395034e0c8288ed78a863662c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 46821 } } }
I20260929 17:58:17.854934 15215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fbcb6f395034e0c8288ed78a863662c [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:17.854934 15214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fbcb6f395034e0c8288ed78a863662c [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:17.855432 15222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:58:17.859287 15222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:58:17.866497 15222 catalog_manager.cc:1347] Generated new cluster ID: 8e23aeda1d78484fba425732a71ed42c
I20260929 17:58:17.866611 15222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:58:17.880805 15222 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:58:17.881968 15222 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:58:17.889098 15222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fbcb6f395034e0c8288ed78a863662c: Generated new TSK 0
I20260929 17:58:17.890096 15222 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.7.118.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.118.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.118.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.1.keytab.
Entry for principal oryx/127.7.118.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.1.keytab.
I20260929 17:58:17.931190 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:0
--local_ip_for_outbound_sockets=127.7.118.1
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:46821
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.1.keytab
--principal=oryx/127.7.118.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:18.068989 15234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:18.069381 15234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:18.069475 15234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:18.074010 15234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:18.074144 15234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:18.074226 15234 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 17:58:18.074317 15234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 17:58:18.080425 15234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.1.keytab
--principal=oryx/127.7.118.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.118.62:46821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:18.081817 15234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:18.083127 15234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:18.095394 15242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:18.095427 15240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:18.095479 15234 server_base.cc:1034] running on GCE node
W20260929 17:58:18.095417 15239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:18.096570 15234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:18.097424 15234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:18.098621 15234 hybrid_clock.cc:648] HybridClock initialized: now 1790704698098602 us; error 61 us; skew 500 ppm
Sep 29 17:58:18 dist-test-slave-r7ph krb5kdc[15117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704698, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:18.106950 15234 init.cc:377] Logged in from keytab as oryx/127.7.118.1@KRBTEST.COM (short username oryx)
I20260929 17:58:18.108074 15234 webserver.cc:460] Webserver started at http://127.7.118.1:45067/ using document root <none> and password file <none>
I20260929 17:58:18.109022 15234 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:18.109174 15234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:18.109472 15234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:18.112200 15234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-0/data/instance:
uuid: "b73665a33caa4f5db919e1740a84f386"
format_stamp: "Formatted at 2026-09-29 17:58:18 on dist-test-slave-r7ph"
server_key: "f32744c57c63da06589545ea09e8d771"
server_key_iv: "266335e52228a5f70129f1eeb787a165"
server_key_version: "encryptionkey@0"
I20260929 17:58:18.112941 15234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance:
uuid: "b73665a33caa4f5db919e1740a84f386"
format_stamp: "Formatted at 2026-09-29 17:58:18 on dist-test-slave-r7ph"
server_key: "f32744c57c63da06589545ea09e8d771"
server_key_iv: "266335e52228a5f70129f1eeb787a165"
server_key_version: "encryptionkey@0"
I20260929 17:58:18.117223 15234 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 17:58:18.120702 15249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:18.122362 15234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:58:18.122550 15234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "b73665a33caa4f5db919e1740a84f386"
format_stamp: "Formatted at 2026-09-29 17:58:18 on dist-test-slave-r7ph"
server_key: "f32744c57c63da06589545ea09e8d771"
server_key_iv: "266335e52228a5f70129f1eeb787a165"
server_key_version: "encryptionkey@0"
I20260929 17:58:18.122694 15234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:18.150391 15234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:18.154402 15234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:18.154635 15234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:18.156780 15234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:18.156865 15234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:18.156935 15234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:18.156986 15234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 17:58:18 dist-test-slave-r7ph krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704698, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM
I20260929 17:58:18.169917 15234 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:45829
I20260929 17:58:18.169909 15364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:45829 every 8 connection(s)
I20260929 17:58:18.171277 15234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 17:58:18.176056 15211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:18.158128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:35667 (local address 127.7.118.62:46821)
0929 17:58:18.158321 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:18.158325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:18.160027 (+ 1702us) server_negotiation.cc:407] Connection header received
0929 17:58:18.161796 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:18.161800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:18.161894 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:18.162006 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:18.163920 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.164625 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.165998 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.166309 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.166432 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:18.170467 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:18.170476 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:18.170479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:18.170532 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:18.172968 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:18.173917 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:18.173925 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:18.173929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:18.174091 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:18.174637 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:18.174640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:18.174642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:18.174920 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:18.175025 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:18.175695 (+ 670us) server_negotiation.cc:299] Negotiation successful
0929 17:58:18.175843 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 17:58:18.177196 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:18.158982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:46821 (local address 127.7.118.1:35667)
0929 17:58:18.159773 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:18.159864 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:18.161481 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:18.162310 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:18.162323 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:18.162801 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:18.163702 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:18.163736 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.164779 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.164782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:18.165846 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:18.165857 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.166497 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.166502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:18.166719 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:18.167445 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:18.167472 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:18.170049 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:18.173163 (+ 3114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:18.173178 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:18.173229 (+ 51us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:18.173711 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:18.174222 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:18.174225 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:18.174227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:18.174473 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:18.175064 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:18.175071 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:18.175377 (+ 306us) client_negotiation.cc:769] Sending connection context
0929 17:58:18.175571 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 17:58:18.176074 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":180,"threads_started":1}
I20260929 17:58:18.179147 15365 heartbeater.cc:347] Connected to a master server at 127.7.118.62:46821
I20260929 17:58:18.179345 15365 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:18.180066 15365 heartbeater.cc:511] Master 127.7.118.62:46821 requested a full tablet report, sending...
I20260929 17:58:18.180208 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 15234
I20260929 17:58:18.180359 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance
I20260929 17:58:18.180755 7640 external_mini_cluster.cc:1428] Setting key d90d6eef5649f02c72bf6fc023c2fd5b
I20260929 17:58:18.182099 15158 ts_manager.cc:194] Registered new tserver with Master: b73665a33caa4f5db919e1740a84f386 (127.7.118.1:45829)
I20260929 17:58:18.183895 15158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.118.1@KRBTEST.COM'} at 127.7.118.1:35667
I20260929 17:58:18.192570 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:18.182061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:46821 (local address 127.7.118.1:54139)
0929 17:58:18.182570 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:18.182586 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:18.182693 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:18.183025 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:18.183029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:18.183281 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:18.183414 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:18.183422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.184580 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.184584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:18.185300 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:18.185312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.186039 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.186043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:18.186225 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:18.186928 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:18.186944 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:18.187364 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:18.190403 (+ 3039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:18.190406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:18.190408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:18.190665 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:18.191106 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:18.191110 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:18.191111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:18.191226 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:18.191822 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:18.191825 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:18.192091 (+ 266us) client_negotiation.cc:769] Sending connection context
0929 17:58:18.192231 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 17:58:18.192391 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":131,"threads_started":1}
I20260929 17:58:18.192731 15211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:18.182097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:54139 (local address 127.7.118.62:46821)
0929 17:58:18.182269 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:18.182275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:18.182666 (+ 391us) server_negotiation.cc:407] Connection header received
0929 17:58:18.182780 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:18.182784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:18.182836 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:18.182929 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:18.183586 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.184438 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.185452 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.185777 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.185861 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:18.187588 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:18.187598 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:18.187601 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:18.187631 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:18.190165 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:18.190799 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:18.190807 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:18.190811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:18.190931 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:18.191334 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:18.191338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:18.191339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:18.191647 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:18.191779 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:18.192331 (+ 552us) server_negotiation.cc:299] Negotiation successful
0929 17:58:18.192498 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":50}
WARNING: no policy specified for oryx/127.7.118.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.118.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.118.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.2.keytab.
Entry for principal oryx/127.7.118.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.2.keytab.
I20260929 17:58:18.222347 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:0
--local_ip_for_outbound_sockets=127.7.118.2
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:46821
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.2.keytab
--principal=oryx/127.7.118.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:18.361717 15374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:18.362062 15374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:18.362120 15374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:18.367225 15374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:18.367339 15374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:18.367388 15374 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 17:58:18.367427 15374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 17:58:18.374049 15374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.2.keytab
--principal=oryx/127.7.118.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.118.62:46821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:18.375739 15374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:18.377475 15374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:18.389943 15380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:18.390362 15379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:18.390450 15382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:18.390472 15374 server_base.cc:1034] running on GCE node
I20260929 17:58:18.391521 15374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:18.392354 15374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:18.395757 15374 hybrid_clock.cc:648] HybridClock initialized: now 1790704698395903 us; error 213 us; skew 500 ppm
Sep 29 17:58:18 dist-test-slave-r7ph krb5kdc[15117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704698, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:18.399847 15374 init.cc:377] Logged in from keytab as oryx/127.7.118.2@KRBTEST.COM (short username oryx)
I20260929 17:58:18.400880 15374 webserver.cc:460] Webserver started at http://127.7.118.2:43959/ using document root <none> and password file <none>
I20260929 17:58:18.401477 15374 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:18.401551 15374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:18.401731 15374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:18.404331 15374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-1/data/instance:
uuid: "22339151149f40f38aa45467baca4537"
format_stamp: "Formatted at 2026-09-29 17:58:18 on dist-test-slave-r7ph"
server_key: "55a51e980fbfe5679e5085f534f36fce"
server_key_iv: "68cfd593ba0a3d338647c69aeeceb69d"
server_key_version: "encryptionkey@0"
I20260929 17:58:18.404950 15374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance:
uuid: "22339151149f40f38aa45467baca4537"
format_stamp: "Formatted at 2026-09-29 17:58:18 on dist-test-slave-r7ph"
server_key: "55a51e980fbfe5679e5085f534f36fce"
server_key_iv: "68cfd593ba0a3d338647c69aeeceb69d"
server_key_version: "encryptionkey@0"
I20260929 17:58:18.409547 15374 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 17:58:18.412817 15389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:18.414997 15374 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 17:58:18.415280 15374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "22339151149f40f38aa45467baca4537"
format_stamp: "Formatted at 2026-09-29 17:58:18 on dist-test-slave-r7ph"
server_key: "55a51e980fbfe5679e5085f534f36fce"
server_key_iv: "68cfd593ba0a3d338647c69aeeceb69d"
server_key_version: "encryptionkey@0"
I20260929 17:58:18.415462 15374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:18.456243 15374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:18.459869 15374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:18.460147 15374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:18.461783 15374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:18.461902 15374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:18.461983 15374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:18.462042 15374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 17:58:18 dist-test-slave-r7ph krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704698, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM
I20260929 17:58:18.473702 15374 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:42039
I20260929 17:58:18.473773 15504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:42039 every 8 connection(s)
I20260929 17:58:18.475106 15374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 17:58:18.478451 15211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:18.462988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:45641 (local address 127.7.118.62:46821)
0929 17:58:18.463135 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:18.463139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:18.464415 (+ 1276us) server_negotiation.cc:407] Connection header received
0929 17:58:18.465680 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:18.465683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:18.465748 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:18.465854 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:18.467770 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.468300 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.469722 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.469966 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.470094 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:18.473214 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:18.473217 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:18.473220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:18.473249 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:18.475441 (+ 2192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:18.476216 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:18.476223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:18.476229 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:18.476325 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:18.477113 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:18.477117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:18.477119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:18.477381 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:18.477509 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:18.478074 (+ 565us) server_negotiation.cc:299] Negotiation successful
0929 17:58:18.478223 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 17:58:18.479383 15398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:18.463568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:46821 (local address 127.7.118.2:45641)
0929 17:58:18.464178 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:18.464231 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:18.465409 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:18.466091 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:18.466106 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:18.466721 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:18.467557 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:18.467584 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.468456 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.468460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:18.469572 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:18.469583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.470159 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.470163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:18.470455 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:18.471279 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:18.471297 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:18.472987 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:18.475651 (+ 2664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:18.475658 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:18.475677 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:18.476098 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:18.476470 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:18.476473 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:18.476475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:18.476944 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:18.477506 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:18.477514 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:18.477805 (+ 291us) client_negotiation.cc:769] Sending connection context
0929 17:58:18.478027 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 17:58:18.478447 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":162,"threads_started":1}
I20260929 17:58:18.480824 15505 heartbeater.cc:347] Connected to a master server at 127.7.118.62:46821
I20260929 17:58:18.480991 15505 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:18.481137 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 15374
I20260929 17:58:18.481261 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance
I20260929 17:58:18.481567 7640 external_mini_cluster.cc:1428] Setting key 7f8f34b22595cf4db47aafdf1ed945e4
I20260929 17:58:18.481681 15505 heartbeater.cc:511] Master 127.7.118.62:46821 requested a full tablet report, sending...
I20260929 17:58:18.482947 15158 ts_manager.cc:194] Registered new tserver with Master: 22339151149f40f38aa45467baca4537 (127.7.118.2:42039)
I20260929 17:58:18.483732 15158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.118.2@KRBTEST.COM'} at 127.7.118.2:45641
I20260929 17:58:18.493054 15511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:18.483606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:46821 (local address 127.7.118.2:39623)
0929 17:58:18.483938 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:18.483950 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:18.484101 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:18.484405 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:18.484408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:18.484663 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:18.484793 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:18.484801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.485864 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.485867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:18.486804 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:18.486815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.487709 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.487712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:18.487804 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:18.488513 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:18.488531 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:18.488903 (+ 372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:18.491315 (+ 2412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:18.491318 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:18.491320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:18.491580 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:18.491925 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:18.491927 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:18.491928 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:18.492019 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:18.492591 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:18.492593 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:18.492682 (+ 89us) client_negotiation.cc:769] Sending connection context
0929 17:58:18.492767 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 17:58:18.492894 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for oryx/127.7.118.3@KRBTEST.COM; defaulting to no policy
I20260929 17:58:18.493185 15211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:18.483646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:39623 (local address 127.7.118.62:46821)
0929 17:58:18.483773 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:18.483778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:18.484052 (+ 274us) server_negotiation.cc:407] Connection header received
0929 17:58:18.484212 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:18.484217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:18.484274 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:18.484366 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:18.484944 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.485683 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.487092 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.487514 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.487669 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:18.489189 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:18.489195 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:18.489202 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:18.489242 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:18.491156 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:18.491701 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:18.491705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:18.491707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:18.491811 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:18.492117 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:18.492120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:18.492123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:18.492477 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:18.492599 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:18.492868 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 17:58:18.492996 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.118.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.118.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.3.keytab.
Entry for principal oryx/127.7.118.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.3.keytab.
I20260929 17:58:18.520923 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:0
--local_ip_for_outbound_sockets=127.7.118.3
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:46821
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.3.keytab
--principal=oryx/127.7.118.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:18.654706 15514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:18.655031 15514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:18.655081 15514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:18.660341 15514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:18.660446 15514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:18.660501 15514 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 17:58:18.660538 15514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 17:58:18.666468 15514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.3.keytab
--principal=oryx/127.7.118.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.118.62:46821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:18.667649 15514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:18.668959 15514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:18.682777 15519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:18.682773 15522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:18.682773 15520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:18.683686 15514 server_base.cc:1034] running on GCE node
I20260929 17:58:18.684265 15514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:18.685276 15514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:18.686501 15514 hybrid_clock.cc:648] HybridClock initialized: now 1790704698686478 us; error 39 us; skew 500 ppm
Sep 29 17:58:18 dist-test-slave-r7ph krb5kdc[15117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704698, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:18.691220 15514 init.cc:377] Logged in from keytab as oryx/127.7.118.3@KRBTEST.COM (short username oryx)
I20260929 17:58:18.692309 15514 webserver.cc:460] Webserver started at http://127.7.118.3:44065/ using document root <none> and password file <none>
I20260929 17:58:18.692967 15514 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:18.693071 15514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:18.693332 15514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:18.695700 15514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-2/data/instance:
uuid: "7a12a5e9fdae4cceabe756a3e18a2037"
format_stamp: "Formatted at 2026-09-29 17:58:18 on dist-test-slave-r7ph"
server_key: "02f99f4e86a375d664c6a2931fe06a91"
server_key_iv: "90e558a386c384f4d0bb7bd3099a8617"
server_key_version: "encryptionkey@0"
I20260929 17:58:18.696380 15514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance:
uuid: "7a12a5e9fdae4cceabe756a3e18a2037"
format_stamp: "Formatted at 2026-09-29 17:58:18 on dist-test-slave-r7ph"
server_key: "02f99f4e86a375d664c6a2931fe06a91"
server_key_iv: "90e558a386c384f4d0bb7bd3099a8617"
server_key_version: "encryptionkey@0"
I20260929 17:58:18.701020 15514 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:58:18.704262 15529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:18.705979 15514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:58:18.706190 15514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "7a12a5e9fdae4cceabe756a3e18a2037"
format_stamp: "Formatted at 2026-09-29 17:58:18 on dist-test-slave-r7ph"
server_key: "02f99f4e86a375d664c6a2931fe06a91"
server_key_iv: "90e558a386c384f4d0bb7bd3099a8617"
server_key_version: "encryptionkey@0"
I20260929 17:58:18.706398 15514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:18.723747 15514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:18.727864 15514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:18.728099 15514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:18.729777 15514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:18.729836 15514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:18.729887 15514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:18.729939 15514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 17:58:18 dist-test-slave-r7ph krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704698, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM
I20260929 17:58:18.742113 15514 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:42025
I20260929 17:58:18.742195 15644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:42025 every 8 connection(s)
I20260929 17:58:18.743331 15514 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 17:58:18.745733 15211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:18.730822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:34447 (local address 127.7.118.62:46821)
0929 17:58:18.731007 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:18.731011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:18.732240 (+ 1229us) server_negotiation.cc:407] Connection header received
0929 17:58:18.733520 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:18.733524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:18.733588 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:18.733670 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:18.735375 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.735986 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.737220 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.737626 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.737753 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:18.741291 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:18.741295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:18.741297 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:18.741328 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:18.743051 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:18.743799 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:18.743803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:18.743805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:18.743945 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:18.744596 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:18.744599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:18.744600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:18.744821 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:18.744950 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:18.745475 (+ 525us) server_negotiation.cc:299] Negotiation successful
0929 17:58:18.745595 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260929 17:58:18.746868 15538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:18.731465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:46821 (local address 127.7.118.3:34447)
0929 17:58:18.732041 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:18.732091 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:18.733165 (+ 1074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:18.733856 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:18.733864 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:18.734474 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:18.735198 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:18.735224 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.736116 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.736120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:18.737069 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:18.737079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.737824 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.737828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:18.738092 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:18.738885 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:18.738907 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:18.741044 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:18.743199 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:18.743205 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:18.743229 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:18.743602 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:18.744081 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:18.744084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:18.744086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:18.744279 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:18.744949 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:18.744955 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:18.745228 (+ 273us) client_negotiation.cc:769] Sending connection context
0929 17:58:18.745403 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 17:58:18.745698 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":138,"threads_started":1}
I20260929 17:58:18.748462 15645 heartbeater.cc:347] Connected to a master server at 127.7.118.62:46821
I20260929 17:58:18.748651 15645 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:18.749267 15645 heartbeater.cc:511] Master 127.7.118.62:46821 requested a full tablet report, sending...
I20260929 17:58:18.749541 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 15514
I20260929 17:58:18.749701 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance
I20260929 17:58:18.750065 7640 external_mini_cluster.cc:1428] Setting key 28d3b564ac895ffc4eec88b935ca40bb
I20260929 17:58:18.750746 15158 ts_manager.cc:194] Registered new tserver with Master: 7a12a5e9fdae4cceabe756a3e18a2037 (127.7.118.3:42025)
I20260929 17:58:18.751513 15158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.118.3@KRBTEST.COM'} at 127.7.118.3:34447
I20260929 17:58:18.753463 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:58:18.760430 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:18.751267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:46821 (local address 127.7.118.3:40503)
0929 17:58:18.751588 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:18.751604 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:18.751754 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:18.752083 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:18.752087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:18.752333 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:18.752470 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:18.752476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.753954 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.753958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:18.754650 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:18.754660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.755316 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.755320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:18.755421 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:18.756018 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:18.756031 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:18.756386 (+ 355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:18.758722 (+ 2336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:18.758725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:18.758727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:18.758957 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:18.759289 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:18.759292 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:18.759295 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:18.759393 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:18.759825 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:18.759828 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:18.759945 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 17:58:18.760058 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 17:58:18.760231 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":4,"thread_start_us":136,"threads_started":1}
I20260929 17:58:18.760421 15211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:18.751328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:40503 (local address 127.7.118.62:46821)
0929 17:58:18.751476 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:18.751480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:18.751679 (+ 199us) server_negotiation.cc:407] Connection header received
0929 17:58:18.751872 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:18.751876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:18.751972 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:18.752061 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:18.753098 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.753815 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.754935 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.755177 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.755298 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:18.756540 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:18.756544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:18.756547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:18.756580 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:18.758568 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:18.759083 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:18.759087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:18.759089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:18.759190 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:18.759504 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:18.759507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:18.759509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:18.759710 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:18.759836 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:18.760113 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 17:58:18.760228 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 17:58:18 dist-test-slave-r7ph krb5kdc[15117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704698, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 17:58:18 dist-test-slave-r7ph krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:18 dist-test-slave-r7ph krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 17:58:18.782321 15211 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:18.774797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55334 (local address 127.7.118.62:46821)
0929 17:58:18.774949 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:18.774953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:18.774990 (+ 37us) server_negotiation.cc:407] Connection header received
0929 17:58:18.775046 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:18.775050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:18.775107 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:18.775200 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:18.775989 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.776604 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.777608 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.777851 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.777953 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:18.782064 (+ 4111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55334: BlockingRecv error: recv got EOF from 127.0.0.1:55334 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 17:58:18 dist-test-slave-r7ph krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704698, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM
I20260929 17:58:18.794181 15211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:18.784326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55348 (local address 127.7.118.62:46821)
0929 17:58:18.784504 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:18.784509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:18.784637 (+ 128us) server_negotiation.cc:407] Connection header received
0929 17:58:18.784817 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:18.784821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:18.784876 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:18.784977 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:18.785726 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.786343 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.787190 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.787435 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.787561 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:18.790398 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:18.790402 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:18.790404 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:18.790438 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:18.791990 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:18.792591 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:18.792595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:18.792597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:18.792696 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:18.793018 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:18.793022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:18.793023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:18.793412 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:18.793585 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:18.793846 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 17:58:18.793999 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 17:58:18.798326 15162 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:58:18.801776 15162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:58:18.819106 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:18.812314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:45829 (local address 127.0.0.1:57706)
0929 17:58:18.813436 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:18.813498 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:18.813648 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:18.814311 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:18.814317 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:18.814341 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:18.814492 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:18.814499 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.815613 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.815619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:18.817170 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:18.817181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.818602 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.818607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:18.818699 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:18.818724 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 17:58:18.818802 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 17:58:18.818881 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"mutex_wait_us":30,"spinlock_wait_cycles":9216,"thread_start_us":88,"threads_started":1}
I20260929 17:58:18.819119 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:18.812743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57706 (local address 127.7.118.1:45829)
0929 17:58:18.813135 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:18.813142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:18.813808 (+ 666us) server_negotiation.cc:407] Connection header received
0929 17:58:18.813900 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:18.813908 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:18.814117 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:18.814309 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:18.814647 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.815471 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.817335 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.818204 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.818362 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:18.818530 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:18.818860 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 17:58:18.818985 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":73,"threads_started":1}
I20260929 17:58:18.819330 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:18.812557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:42025 (local address 127.0.0.1:40890)
0929 17:58:18.813431 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:18.813498 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:18.813647 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:18.814519 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:18.814523 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:18.814549 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:18.814683 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:18.814717 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.816287 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.816290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:18.817560 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:18.817572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.818962 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.818966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:18.819047 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:18.819067 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 17:58:18.819118 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 17:58:18.819196 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":117,"spinlock_wait_cycles":6784,"thread_start_us":122,"threads_started":1}
I20260929 17:58:18.819826 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:18.813548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37688 (local address 127.7.118.2:42039)
0929 17:58:18.813955 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:18.813967 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:18.813990 (+ 23us) server_negotiation.cc:407] Connection header received
0929 17:58:18.814075 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:18.814080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:18.814314 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:18.814497 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:18.814757 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.815988 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.817464 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.818149 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.818313 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:18.818520 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:18.818975 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 17:58:18.819070 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":106,"threads_started":1}
I20260929 17:58:18.820304 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:18.812988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40890 (local address 127.7.118.3:42025)
0929 17:58:18.813982 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:18.813993 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:18.814012 (+ 19us) server_negotiation.cc:407] Connection header received
0929 17:58:18.814075 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:18.814080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:18.814321 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:18.814881 (+ 560us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:18.815028 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.816168 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.817692 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.818532 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.819763 (+ 1231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:18.819994 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:18.820046 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 17:58:18.820152 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":902,"thread_start_us":124,"threads_started":1}
I20260929 17:58:18.820564 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:18.813088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:42039 (local address 127.0.0.1:37688)
0929 17:58:18.813749 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:18.813763 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:18.813839 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:18.814492 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:18.814495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:18.814510 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:18.814634 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:18.814641 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.816128 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.816131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:18.817315 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:18.817326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.818322 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.818326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:18.818459 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:18.818506 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 17:58:18.818660 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 17:58:18.818828 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":4,"spinlock_wait_cycles":9344,"thread_start_us":88,"threads_started":1}
I20260929 17:58:18.822801 15287 tablet_service.cc:1467] Processing CreateTablet for tablet bd8566505fa345f08111dc765a575339 (DEFAULT_TABLE table=test-table [id=55f2d9696f044ee682507ff813c67c37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:58:18.823290 15439 tablet_service.cc:1467] Processing CreateTablet for tablet bd8566505fa345f08111dc765a575339 (DEFAULT_TABLE table=test-table [id=55f2d9696f044ee682507ff813c67c37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:58:18.823288 15579 tablet_service.cc:1467] Processing CreateTablet for tablet bd8566505fa345f08111dc765a575339 (DEFAULT_TABLE table=test-table [id=55f2d9696f044ee682507ff813c67c37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:58:18.824061 15287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd8566505fa345f08111dc765a575339. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:18.824386 15439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd8566505fa345f08111dc765a575339. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:18.824405 15579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd8566505fa345f08111dc765a575339. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:18.833668 15673 tablet_bootstrap.cc:492] T bd8566505fa345f08111dc765a575339 P b73665a33caa4f5db919e1740a84f386: Bootstrap starting.
I20260929 17:58:18.834659 15674 tablet_bootstrap.cc:492] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037: Bootstrap starting.
I20260929 17:58:18.836163 15673 tablet_bootstrap.cc:654] T bd8566505fa345f08111dc765a575339 P b73665a33caa4f5db919e1740a84f386: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:18.837119 15673 log.cc:824] T bd8566505fa345f08111dc765a575339 P b73665a33caa4f5db919e1740a84f386: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:18.837561 15674 tablet_bootstrap.cc:654] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:18.837921 15675 tablet_bootstrap.cc:492] T bd8566505fa345f08111dc765a575339 P 22339151149f40f38aa45467baca4537: Bootstrap starting.
I20260929 17:58:18.838683 15674 log.cc:824] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:18.840205 15675 tablet_bootstrap.cc:654] T bd8566505fa345f08111dc765a575339 P 22339151149f40f38aa45467baca4537: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:18.840312 15673 tablet_bootstrap.cc:492] T bd8566505fa345f08111dc765a575339 P b73665a33caa4f5db919e1740a84f386: No bootstrap required, opened a new log
I20260929 17:58:18.840531 15673 ts_tablet_manager.cc:1397] T bd8566505fa345f08111dc765a575339 P b73665a33caa4f5db919e1740a84f386: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 17:58:18.840780 15674 tablet_bootstrap.cc:492] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037: No bootstrap required, opened a new log
I20260929 17:58:18.840961 15674 ts_tablet_manager.cc:1397] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 17:58:18.840997 15675 log.cc:824] T bd8566505fa345f08111dc765a575339 P 22339151149f40f38aa45467baca4537: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:18.843050 15675 tablet_bootstrap.cc:492] T bd8566505fa345f08111dc765a575339 P 22339151149f40f38aa45467baca4537: No bootstrap required, opened a new log
I20260929 17:58:18.843321 15675 ts_tablet_manager.cc:1397] T bd8566505fa345f08111dc765a575339 P 22339151149f40f38aa45467baca4537: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 17:58:18.845206 15673 raft_consensus.cc:348] T bd8566505fa345f08111dc765a575339 P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } }
I20260929 17:58:18.845191 15674 raft_consensus.cc:348] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } }
I20260929 17:58:18.845518 15673 raft_consensus.cc:374] T bd8566505fa345f08111dc765a575339 P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:18.845520 15674 raft_consensus.cc:374] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:18.845582 15673 raft_consensus.cc:729] T bd8566505fa345f08111dc765a575339 P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b73665a33caa4f5db919e1740a84f386, State: Initialized, Role: FOLLOWER
I20260929 17:58:18.845592 15674 raft_consensus.cc:729] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a12a5e9fdae4cceabe756a3e18a2037, State: Initialized, Role: FOLLOWER
I20260929 17:58:18.846105 15674 consensus_queue.cc:260] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } }
I20260929 17:58:18.846105 15673 consensus_queue.cc:260] T bd8566505fa345f08111dc765a575339 P b73665a33caa4f5db919e1740a84f386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } }
I20260929 17:58:18.846688 15675 raft_consensus.cc:348] T bd8566505fa345f08111dc765a575339 P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } }
I20260929 17:58:18.846980 15675 raft_consensus.cc:374] T bd8566505fa345f08111dc765a575339 P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:18.847064 15675 raft_consensus.cc:729] T bd8566505fa345f08111dc765a575339 P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22339151149f40f38aa45467baca4537, State: Initialized, Role: FOLLOWER
I20260929 17:58:18.847282 15673 ts_tablet_manager.cc:1428] T bd8566505fa345f08111dc765a575339 P b73665a33caa4f5db919e1740a84f386: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260929 17:58:18.847282 15674 ts_tablet_manager.cc:1428] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 17:58:18.847563 15675 consensus_queue.cc:260] T bd8566505fa345f08111dc765a575339 P 22339151149f40f38aa45467baca4537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } }
I20260929 17:58:18.847708 15645 heartbeater.cc:503] Master 127.7.118.62:46821 was elected leader, sending a full tablet report...
I20260929 17:58:18.847879 15365 heartbeater.cc:503] Master 127.7.118.62:46821 was elected leader, sending a full tablet report...
I20260929 17:58:18.848670 15675 ts_tablet_manager.cc:1428] T bd8566505fa345f08111dc765a575339 P 22339151149f40f38aa45467baca4537: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 17:58:18.848932 15505 heartbeater.cc:503] Master 127.7.118.62:46821 was elected leader, sending a full tablet report...
W20260929 17:58:18.922982 15366 tablet.cc:2369] T bd8566505fa345f08111dc765a575339 P b73665a33caa4f5db919e1740a84f386: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:58:18.950662 15680 raft_consensus.cc:482] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:58:18.950948 15680 raft_consensus.cc:504] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } }
I20260929 17:58:18.952204 15680 leader_election.cc:288] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b73665a33caa4f5db919e1740a84f386 (127.7.118.1:45829), 22339151149f40f38aa45467baca4537 (127.7.118.2:42039)
I20260929 17:58:18.957067 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:18.952587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:49455 (local address 127.7.118.2:42039)
0929 17:58:18.952793 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:18.952798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:18.952817 (+ 19us) server_negotiation.cc:407] Connection header received
0929 17:58:18.952996 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:18.952998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:18.953043 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:18.953172 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:18.953500 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.954072 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.955726 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.956428 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.956552 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:18.956621 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:18.956813 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 17:58:18.956902 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260929 17:58:18.957129 15538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:18.952516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:42039 (local address 127.7.118.3:49455)
0929 17:58:18.952746 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:18.952762 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:18.952871 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:18.953180 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:18.953183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:18.953200 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:18.953360 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:18.953386 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.954223 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.954226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:18.955557 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:18.955570 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.956591 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.956594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:18.956682 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:18.956702 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 17:58:18.956773 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 17:58:18.956840 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":4}
I20260929 17:58:18.957548 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:18.952622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:54313 (local address 127.7.118.1:45829)
0929 17:58:18.952750 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:18.952756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:18.953111 (+ 355us) server_negotiation.cc:407] Connection header received
0929 17:58:18.953256 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:18.953261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:18.953319 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:18.953388 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:18.954045 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.954737 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.956276 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.956917 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.957024 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:18.957084 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:18.957353 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 17:58:18.957437 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 17:58:18.957561 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:18.952546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:45829 (local address 127.7.118.3:54313)
0929 17:58:18.953003 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:18.953021 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:18.953101 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:18.953432 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:18.953436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:18.953458 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:18.953917 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:18.953925 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.954856 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.954860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:18.956129 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:18.956140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:18.957067 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:18.957071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:18.957178 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:18.957203 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 17:58:18.957324 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 17:58:18.957424 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":157,"threads_started":1}
I20260929 17:58:18.957970 15459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8566505fa345f08111dc765a575339" candidate_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22339151149f40f38aa45467baca4537" is_pre_election: true
I20260929 17:58:18.958282 15319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8566505fa345f08111dc765a575339" candidate_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b73665a33caa4f5db919e1740a84f386" is_pre_election: true
I20260929 17:58:18.958377 15459 raft_consensus.cc:2454] T bd8566505fa345f08111dc765a575339 P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a12a5e9fdae4cceabe756a3e18a2037 in term 0.
I20260929 17:58:18.958585 15319 raft_consensus.cc:2454] T bd8566505fa345f08111dc765a575339 P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a12a5e9fdae4cceabe756a3e18a2037 in term 0.
I20260929 17:58:18.958972 15531 leader_election.cc:302] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22339151149f40f38aa45467baca4537, 7a12a5e9fdae4cceabe756a3e18a2037; no voters:
I20260929 17:58:18.959298 15680 raft_consensus.cc:2781] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:58:18.959401 15680 raft_consensus.cc:482] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:58:18.959462 15680 raft_consensus.cc:3037] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:18.960537 15680 raft_consensus.cc:504] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } }
I20260929 17:58:18.960929 15680 leader_election.cc:288] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037 [CANDIDATE]: Term 1 election: Requested vote from peers b73665a33caa4f5db919e1740a84f386 (127.7.118.1:45829), 22339151149f40f38aa45467baca4537 (127.7.118.2:42039)
I20260929 17:58:18.961391 15319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8566505fa345f08111dc765a575339" candidate_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b73665a33caa4f5db919e1740a84f386"
I20260929 17:58:18.961437 15459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8566505fa345f08111dc765a575339" candidate_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22339151149f40f38aa45467baca4537"
I20260929 17:58:18.961537 15319 raft_consensus.cc:3037] T bd8566505fa345f08111dc765a575339 P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:18.961604 15459 raft_consensus.cc:3037] T bd8566505fa345f08111dc765a575339 P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:18.962769 15319 raft_consensus.cc:2454] T bd8566505fa345f08111dc765a575339 P b73665a33caa4f5db919e1740a84f386 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a12a5e9fdae4cceabe756a3e18a2037 in term 1.
I20260929 17:58:18.963007 15459 raft_consensus.cc:2454] T bd8566505fa345f08111dc765a575339 P 22339151149f40f38aa45467baca4537 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a12a5e9fdae4cceabe756a3e18a2037 in term 1.
I20260929 17:58:18.963330 15533 leader_election.cc:302] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a12a5e9fdae4cceabe756a3e18a2037, b73665a33caa4f5db919e1740a84f386; no voters:
I20260929 17:58:18.963582 15680 raft_consensus.cc:2781] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:58:18.963856 15680 raft_consensus.cc:686] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 1 LEADER]: Becoming Leader. State: Replica: 7a12a5e9fdae4cceabe756a3e18a2037, State: Running, Role: LEADER
I20260929 17:58:18.964252 15680 consensus_queue.cc:237] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } }
I20260929 17:58:18.967924 15161 catalog_manager.cc:5495] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a12a5e9fdae4cceabe756a3e18a2037 (127.7.118.3). New cstate: current_term: 1 leader_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } health_report { overall_health: UNKNOWN } } }
W20260929 17:58:18.976729 15506 tablet.cc:2369] T bd8566505fa345f08111dc765a575339 P 22339151149f40f38aa45467baca4537: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:58:18.994530 15646 tablet.cc:2369] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:58:19.014549 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:19.009794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40894 (local address 127.7.118.3:42025)
0929 17:58:19.009959 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:19.009963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:19.009978 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:58:19.010221 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:19.010224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:19.010377 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:19.010493 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:58:19.010860 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.011486 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.012756 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.013051 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.013153 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:19.013495 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:58:19.014019 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:58:19.014120 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:19.014240 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 17:58:19.014421 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 17:58:19.022840 15459 raft_consensus.cc:1261] T bd8566505fa345f08111dc765a575339 P 22339151149f40f38aa45467baca4537 [term 1 FOLLOWER]: Refusing update from remote peer 7a12a5e9fdae4cceabe756a3e18a2037: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:58:19.022850 15319 raft_consensus.cc:1261] T bd8566505fa345f08111dc765a575339 P b73665a33caa4f5db919e1740a84f386 [term 1 FOLLOWER]: Refusing update from remote peer 7a12a5e9fdae4cceabe756a3e18a2037: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:58:19.023628 15680 consensus_queue.cc:1035] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:58:19.023852 15683 consensus_queue.cc:1035] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:58:19.037263 15688 mvcc.cc:205] Tried to move back new op lower bound from 7334726447188094976 to 7334726446960754688. Current Snapshot: MvccSnapshot[applied={T|T < 7334726447188094976 or (T in {7334726447188094976})}]
I20260929 17:58:19.045921 15162 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:58:19.046154 15162 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:58:19.049404 15162 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0fbcb6f395034e0c8288ed78a863662c: Sending DeleteTablet for 3 replicas of tablet bd8566505fa345f08111dc765a575339
I20260929 17:58:19.050630 15287 tablet_service.cc:1514] Processing DeleteTablet for tablet bd8566505fa345f08111dc765a575339 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:58:19 UTC) from {username='oryx', principal='oryx/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:57706
I20260929 17:58:19.051316 15579 tablet_service.cc:1514] Processing DeleteTablet for tablet bd8566505fa345f08111dc765a575339 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:58:19 UTC) from {username='oryx', principal='oryx/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:40890
I20260929 17:58:19.051828 15439 tablet_service.cc:1514] Processing DeleteTablet for tablet bd8566505fa345f08111dc765a575339 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:58:19 UTC) from {username='oryx', principal='oryx/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:37688
I20260929 17:58:19.052259 15696 tablet_replica.cc:330] T bd8566505fa345f08111dc765a575339 P b73665a33caa4f5db919e1740a84f386: stopping tablet replica
I20260929 17:58:19.052690 15698 tablet_replica.cc:330] T bd8566505fa345f08111dc765a575339 P 22339151149f40f38aa45467baca4537: stopping tablet replica
I20260929 17:58:19.052973 15696 raft_consensus.cc:2229] T bd8566505fa345f08111dc765a575339 P b73665a33caa4f5db919e1740a84f386 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:58:19.053026 15698 raft_consensus.cc:2229] T bd8566505fa345f08111dc765a575339 P 22339151149f40f38aa45467baca4537 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:58:19.053447 15698 raft_consensus.cc:2258] T bd8566505fa345f08111dc765a575339 P 22339151149f40f38aa45467baca4537 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:58:19.053433 15696 raft_consensus.cc:2258] T bd8566505fa345f08111dc765a575339 P b73665a33caa4f5db919e1740a84f386 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:58:19.053807 15161 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:58:19.054550 15161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:58:19.055768 15698 ts_tablet_manager.cc:1905] T bd8566505fa345f08111dc765a575339 P 22339151149f40f38aa45467baca4537: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:58:19.055835 15696 ts_tablet_manager.cc:1905] T bd8566505fa345f08111dc765a575339 P b73665a33caa4f5db919e1740a84f386: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:58:19.059489 15697 tablet_replica.cc:330] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037: stopping tablet replica
I20260929 17:58:19.059866 15696 ts_tablet_manager.cc:1918] T bd8566505fa345f08111dc765a575339 P b73665a33caa4f5db919e1740a84f386: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:58:19.059863 15698 ts_tablet_manager.cc:1918] T bd8566505fa345f08111dc765a575339 P 22339151149f40f38aa45467baca4537: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:58:19.060050 15698 log.cc:1196] T bd8566505fa345f08111dc765a575339 P 22339151149f40f38aa45467baca4537: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-1/wal/wals/bd8566505fa345f08111dc765a575339
I20260929 17:58:19.060151 15696 log.cc:1196] T bd8566505fa345f08111dc765a575339 P b73665a33caa4f5db919e1740a84f386: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-0/wal/wals/bd8566505fa345f08111dc765a575339
I20260929 17:58:19.060475 15698 ts_tablet_manager.cc:1939] T bd8566505fa345f08111dc765a575339 P 22339151149f40f38aa45467baca4537: Deleting consensus metadata
I20260929 17:58:19.060652 15696 ts_tablet_manager.cc:1939] T bd8566505fa345f08111dc765a575339 P b73665a33caa4f5db919e1740a84f386: Deleting consensus metadata
I20260929 17:58:19.062217 15149 catalog_manager.cc:4841] TS b73665a33caa4f5db919e1740a84f386 (127.7.118.1:45829): tablet bd8566505fa345f08111dc765a575339 (table test-table [id=55f2d9696f044ee682507ff813c67c37]) successfully deleted
I20260929 17:58:19.062463 15697 raft_consensus.cc:2229] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:58:19.062486 15147 catalog_manager.cc:4841] TS 22339151149f40f38aa45467baca4537 (127.7.118.2:42039): tablet bd8566505fa345f08111dc765a575339 (table test-table [id=55f2d9696f044ee682507ff813c67c37]) successfully deleted
I20260929 17:58:19.062943 15697 raft_consensus.cc:2258] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:58:19.063068 15579 tablet_service.cc:1467] Processing CreateTablet for tablet 719f8716016b4924be10a4cb3b8e659e (DEFAULT_TABLE table=test-table [id=22ccf7c815f049f1a32d334ab5de39f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:58:19.063140 15439 tablet_service.cc:1467] Processing CreateTablet for tablet 719f8716016b4924be10a4cb3b8e659e (DEFAULT_TABLE table=test-table [id=22ccf7c815f049f1a32d334ab5de39f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:58:19.063385 15579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 719f8716016b4924be10a4cb3b8e659e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:19.063385 15439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 719f8716016b4924be10a4cb3b8e659e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:19.063799 15287 tablet_service.cc:1467] Processing CreateTablet for tablet 719f8716016b4924be10a4cb3b8e659e (DEFAULT_TABLE table=test-table [id=22ccf7c815f049f1a32d334ab5de39f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:58:19.064069 15287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 719f8716016b4924be10a4cb3b8e659e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:19.065991 15675 tablet_bootstrap.cc:492] T 719f8716016b4924be10a4cb3b8e659e P 22339151149f40f38aa45467baca4537: Bootstrap starting.
I20260929 17:58:19.066017 15674 tablet_bootstrap.cc:492] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037: Bootstrap starting.
I20260929 17:58:19.066469 15673 tablet_bootstrap.cc:492] T 719f8716016b4924be10a4cb3b8e659e P b73665a33caa4f5db919e1740a84f386: Bootstrap starting.
I20260929 17:58:19.067097 15675 tablet_bootstrap.cc:654] T 719f8716016b4924be10a4cb3b8e659e P 22339151149f40f38aa45467baca4537: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:19.067112 15674 tablet_bootstrap.cc:654] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:19.067530 15673 tablet_bootstrap.cc:654] T 719f8716016b4924be10a4cb3b8e659e P b73665a33caa4f5db919e1740a84f386: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:19.069228 15697 ts_tablet_manager.cc:1905] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:58:19.069278 15673 tablet_bootstrap.cc:492] T 719f8716016b4924be10a4cb3b8e659e P b73665a33caa4f5db919e1740a84f386: No bootstrap required, opened a new log
I20260929 17:58:19.069375 15673 ts_tablet_manager.cc:1397] T 719f8716016b4924be10a4cb3b8e659e P b73665a33caa4f5db919e1740a84f386: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 17:58:19.070181 15673 raft_consensus.cc:348] T 719f8716016b4924be10a4cb3b8e659e P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.070374 15673 raft_consensus.cc:374] T 719f8716016b4924be10a4cb3b8e659e P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:19.070415 15673 raft_consensus.cc:729] T 719f8716016b4924be10a4cb3b8e659e P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b73665a33caa4f5db919e1740a84f386, State: Initialized, Role: FOLLOWER
I20260929 17:58:19.070544 15673 consensus_queue.cc:260] T 719f8716016b4924be10a4cb3b8e659e P b73665a33caa4f5db919e1740a84f386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.070778 15675 tablet_bootstrap.cc:492] T 719f8716016b4924be10a4cb3b8e659e P 22339151149f40f38aa45467baca4537: No bootstrap required, opened a new log
I20260929 17:58:19.070801 15673 ts_tablet_manager.cc:1428] T 719f8716016b4924be10a4cb3b8e659e P b73665a33caa4f5db919e1740a84f386: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 17:58:19.070905 15675 ts_tablet_manager.cc:1397] T 719f8716016b4924be10a4cb3b8e659e P 22339151149f40f38aa45467baca4537: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 17:58:19.071455 15675 raft_consensus.cc:348] T 719f8716016b4924be10a4cb3b8e659e P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.071611 15675 raft_consensus.cc:374] T 719f8716016b4924be10a4cb3b8e659e P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:19.071652 15675 raft_consensus.cc:729] T 719f8716016b4924be10a4cb3b8e659e P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22339151149f40f38aa45467baca4537, State: Initialized, Role: FOLLOWER
I20260929 17:58:19.071795 15675 consensus_queue.cc:260] T 719f8716016b4924be10a4cb3b8e659e P 22339151149f40f38aa45467baca4537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.072085 15675 ts_tablet_manager.cc:1428] T 719f8716016b4924be10a4cb3b8e659e P 22339151149f40f38aa45467baca4537: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:58:19.073360 15697 ts_tablet_manager.cc:1918] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:58:19.073547 15697 log.cc:1196] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-2/wal/wals/bd8566505fa345f08111dc765a575339
I20260929 17:58:19.074020 15697 ts_tablet_manager.cc:1939] T bd8566505fa345f08111dc765a575339 P 7a12a5e9fdae4cceabe756a3e18a2037: Deleting consensus metadata
I20260929 17:58:19.074476 15674 tablet_bootstrap.cc:492] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037: No bootstrap required, opened a new log
I20260929 17:58:19.074605 15674 ts_tablet_manager.cc:1397] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260929 17:58:19.075124 15674 raft_consensus.cc:348] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.075256 15674 raft_consensus.cc:374] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:19.075297 15674 raft_consensus.cc:729] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a12a5e9fdae4cceabe756a3e18a2037, State: Initialized, Role: FOLLOWER
I20260929 17:58:19.075459 15674 consensus_queue.cc:260] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.075809 15674 ts_tablet_manager.cc:1428] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 17:58:19.076038 15147 catalog_manager.cc:4841] TS 7a12a5e9fdae4cceabe756a3e18a2037 (127.7.118.3:42025): tablet bd8566505fa345f08111dc765a575339 (table test-table [id=55f2d9696f044ee682507ff813c67c37]) successfully deleted
I20260929 17:58:19.208199 15693 raft_consensus.cc:482] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:58:19.208351 15693 raft_consensus.cc:504] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.208901 15693 leader_election.cc:288] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b73665a33caa4f5db919e1740a84f386 (127.7.118.1:45829), 22339151149f40f38aa45467baca4537 (127.7.118.2:42039)
I20260929 17:58:19.209426 15319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719f8716016b4924be10a4cb3b8e659e" candidate_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b73665a33caa4f5db919e1740a84f386" is_pre_election: true
I20260929 17:58:19.209460 15459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719f8716016b4924be10a4cb3b8e659e" candidate_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22339151149f40f38aa45467baca4537" is_pre_election: true
I20260929 17:58:19.209609 15319 raft_consensus.cc:2454] T 719f8716016b4924be10a4cb3b8e659e P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a12a5e9fdae4cceabe756a3e18a2037 in term 0.
I20260929 17:58:19.209637 15459 raft_consensus.cc:2454] T 719f8716016b4924be10a4cb3b8e659e P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a12a5e9fdae4cceabe756a3e18a2037 in term 0.
I20260929 17:58:19.209969 15531 leader_election.cc:302] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22339151149f40f38aa45467baca4537, 7a12a5e9fdae4cceabe756a3e18a2037; no voters:
I20260929 17:58:19.210188 15693 raft_consensus.cc:2781] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:58:19.210285 15693 raft_consensus.cc:482] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:58:19.210336 15693 raft_consensus.cc:3037] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:19.211447 15693 raft_consensus.cc:504] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.211872 15693 leader_election.cc:288] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037 [CANDIDATE]: Term 1 election: Requested vote from peers b73665a33caa4f5db919e1740a84f386 (127.7.118.1:45829), 22339151149f40f38aa45467baca4537 (127.7.118.2:42039)
I20260929 17:58:19.212297 15319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719f8716016b4924be10a4cb3b8e659e" candidate_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b73665a33caa4f5db919e1740a84f386"
I20260929 17:58:19.212363 15459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719f8716016b4924be10a4cb3b8e659e" candidate_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22339151149f40f38aa45467baca4537"
I20260929 17:58:19.212446 15319 raft_consensus.cc:3037] T 719f8716016b4924be10a4cb3b8e659e P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:19.212563 15459 raft_consensus.cc:3037] T 719f8716016b4924be10a4cb3b8e659e P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:19.213449 15319 raft_consensus.cc:2454] T 719f8716016b4924be10a4cb3b8e659e P b73665a33caa4f5db919e1740a84f386 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a12a5e9fdae4cceabe756a3e18a2037 in term 1.
I20260929 17:58:19.213869 15533 leader_election.cc:302] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a12a5e9fdae4cceabe756a3e18a2037, b73665a33caa4f5db919e1740a84f386; no voters:
I20260929 17:58:19.213898 15459 raft_consensus.cc:2454] T 719f8716016b4924be10a4cb3b8e659e P 22339151149f40f38aa45467baca4537 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a12a5e9fdae4cceabe756a3e18a2037 in term 1.
I20260929 17:58:19.214107 15693 raft_consensus.cc:2781] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:58:19.214223 15693 raft_consensus.cc:686] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037 [term 1 LEADER]: Becoming Leader. State: Replica: 7a12a5e9fdae4cceabe756a3e18a2037, State: Running, Role: LEADER
I20260929 17:58:19.214404 15693 consensus_queue.cc:237] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.216377 15161 catalog_manager.cc:5495] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a12a5e9fdae4cceabe756a3e18a2037 (127.7.118.3). New cstate: current_term: 1 leader_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } health_report { overall_health: HEALTHY } } }
Sep 29 17:58:19 dist-test-slave-r7ph krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704697, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.62@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM
I20260929 17:58:19.258059 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:19.248153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:46821 (local address 127.0.0.1:55354)
0929 17:58:19.248512 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:19.248536 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:19.248682 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:19.248951 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:19.248954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:19.249511 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:19.249610 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:19.249617 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.250438 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.250444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:19.250998 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:19.251006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.251709 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.251712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:19.251809 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:19.252443 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:19.252455 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:19.254016 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:19.256262 (+ 2246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:19.256273 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:19.256285 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:19.256587 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:19.256902 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:19.256905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:19.256906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:19.257005 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:19.257519 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:19.257527 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:19.257653 (+ 126us) client_negotiation.cc:769] Sending connection context
0929 17:58:19.257767 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 17:58:19.257908 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":159,"threads_started":1}
I20260929 17:58:19.258215 15211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:19.248297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55354 (local address 127.7.118.62:46821)
0929 17:58:19.248473 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:19.248477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:19.248612 (+ 135us) server_negotiation.cc:407] Connection header received
0929 17:58:19.248791 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:19.248794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:19.248835 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:19.248933 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:19.249734 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.250335 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.251271 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.251545 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.251699 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:19.254294 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:19.254299 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:19.254301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:19.254333 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:19.256092 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:19.256712 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:19.256715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:19.256716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:19.256801 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:19.257132 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:19.257135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:19.257136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:19.257343 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:19.257447 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:19.257869 (+ 422us) server_negotiation.cc:299] Negotiation successful
0929 17:58:19.258000 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 17:58:19.262485 15161 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:55354:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 17:58:19.263252 15161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:58:19.269068 15439 tablet_service.cc:1467] Processing CreateTablet for tablet 74fc73caa0cc464794bc4ad6f9df36cc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6a0772ac7a244d6fb6137895dd411a6b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 17:58:19.269013 15287 tablet_service.cc:1467] Processing CreateTablet for tablet 74fc73caa0cc464794bc4ad6f9df36cc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6a0772ac7a244d6fb6137895dd411a6b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 17:58:19.269095 15579 tablet_service.cc:1467] Processing CreateTablet for tablet 74fc73caa0cc464794bc4ad6f9df36cc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6a0772ac7a244d6fb6137895dd411a6b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 17:58:19.269454 15287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74fc73caa0cc464794bc4ad6f9df36cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:19.269454 15439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74fc73caa0cc464794bc4ad6f9df36cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:19.269546 15579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74fc73caa0cc464794bc4ad6f9df36cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:19.272158 15674 tablet_bootstrap.cc:492] T 74fc73caa0cc464794bc4ad6f9df36cc P 7a12a5e9fdae4cceabe756a3e18a2037: Bootstrap starting.
I20260929 17:58:19.272503 15675 tablet_bootstrap.cc:492] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537: Bootstrap starting.
I20260929 17:58:19.272790 15673 tablet_bootstrap.cc:492] T 74fc73caa0cc464794bc4ad6f9df36cc P b73665a33caa4f5db919e1740a84f386: Bootstrap starting.
I20260929 17:58:19.273209 15674 tablet_bootstrap.cc:654] T 74fc73caa0cc464794bc4ad6f9df36cc P 7a12a5e9fdae4cceabe756a3e18a2037: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:19.273556 15675 tablet_bootstrap.cc:654] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:19.273779 15673 tablet_bootstrap.cc:654] T 74fc73caa0cc464794bc4ad6f9df36cc P b73665a33caa4f5db919e1740a84f386: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:19.275118 15674 tablet_bootstrap.cc:492] T 74fc73caa0cc464794bc4ad6f9df36cc P 7a12a5e9fdae4cceabe756a3e18a2037: No bootstrap required, opened a new log
I20260929 17:58:19.275153 15675 tablet_bootstrap.cc:492] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537: No bootstrap required, opened a new log
I20260929 17:58:19.275245 15675 ts_tablet_manager.cc:1397] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 17:58:19.275249 15674 ts_tablet_manager.cc:1397] T 74fc73caa0cc464794bc4ad6f9df36cc P 7a12a5e9fdae4cceabe756a3e18a2037: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 17:58:19.275281 15673 tablet_bootstrap.cc:492] T 74fc73caa0cc464794bc4ad6f9df36cc P b73665a33caa4f5db919e1740a84f386: No bootstrap required, opened a new log
I20260929 17:58:19.275369 15673 ts_tablet_manager.cc:1397] T 74fc73caa0cc464794bc4ad6f9df36cc P b73665a33caa4f5db919e1740a84f386: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 17:58:19.275777 15674 raft_consensus.cc:348] T 74fc73caa0cc464794bc4ad6f9df36cc P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.275789 15673 raft_consensus.cc:348] T 74fc73caa0cc464794bc4ad6f9df36cc P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.275789 15675 raft_consensus.cc:348] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.275907 15675 raft_consensus.cc:374] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:19.275907 15673 raft_consensus.cc:374] T 74fc73caa0cc464794bc4ad6f9df36cc P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:19.275921 15674 raft_consensus.cc:374] T 74fc73caa0cc464794bc4ad6f9df36cc P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:19.275966 15674 raft_consensus.cc:729] T 74fc73caa0cc464794bc4ad6f9df36cc P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a12a5e9fdae4cceabe756a3e18a2037, State: Initialized, Role: FOLLOWER
I20260929 17:58:19.275965 15673 raft_consensus.cc:729] T 74fc73caa0cc464794bc4ad6f9df36cc P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b73665a33caa4f5db919e1740a84f386, State: Initialized, Role: FOLLOWER
I20260929 17:58:19.275962 15675 raft_consensus.cc:729] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22339151149f40f38aa45467baca4537, State: Initialized, Role: FOLLOWER
I20260929 17:58:19.276098 15674 consensus_queue.cc:260] T 74fc73caa0cc464794bc4ad6f9df36cc P 7a12a5e9fdae4cceabe756a3e18a2037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.276098 15673 consensus_queue.cc:260] T 74fc73caa0cc464794bc4ad6f9df36cc P b73665a33caa4f5db919e1740a84f386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.276098 15675 consensus_queue.cc:260] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.276362 15674 ts_tablet_manager.cc:1428] T 74fc73caa0cc464794bc4ad6f9df36cc P 7a12a5e9fdae4cceabe756a3e18a2037: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:58:19.276377 15675 ts_tablet_manager.cc:1428] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 17:58:19.276404 15673 ts_tablet_manager.cc:1428] T 74fc73caa0cc464794bc4ad6f9df36cc P b73665a33caa4f5db919e1740a84f386: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:58:19.276623 15679 tablet_replica.cc:439] T 74fc73caa0cc464794bc4ad6f9df36cc P b73665a33caa4f5db919e1740a84f386: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } } }
I20260929 17:58:19.276611 15681 tablet_replica.cc:439] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } } }
I20260929 17:58:19.276777 15681 tablet_replica.cc:442] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:58:19.276777 15679 tablet_replica.cc:442] T 74fc73caa0cc464794bc4ad6f9df36cc P b73665a33caa4f5db919e1740a84f386: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:58:19.276964 15680 tablet_replica.cc:439] T 74fc73caa0cc464794bc4ad6f9df36cc P 7a12a5e9fdae4cceabe756a3e18a2037: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } } }
I20260929 17:58:19.277100 15680 tablet_replica.cc:442] T 74fc73caa0cc464794bc4ad6f9df36cc P 7a12a5e9fdae4cceabe756a3e18a2037: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:58:19.310015 15681 raft_consensus.cc:482] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:58:19.310165 15681 raft_consensus.cc:504] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.311470 15681 leader_election.cc:288] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b73665a33caa4f5db919e1740a84f386 (127.7.118.1:45829), 7a12a5e9fdae4cceabe756a3e18a2037 (127.7.118.3:42025)
I20260929 17:58:19.315670 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:19.311791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:50187 (local address 127.7.118.1:45829)
0929 17:58:19.311935 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:19.311941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:19.312185 (+ 244us) server_negotiation.cc:407] Connection header received
0929 17:58:19.312356 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:19.312359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:19.312410 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:19.312503 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:19.312814 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.313328 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.314652 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.315095 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.315218 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:19.315284 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:19.315473 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 17:58:19.315546 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 17:58:19.315665 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:19.311861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:39705 (local address 127.7.118.3:42025)
0929 17:58:19.312003 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:19.312009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:19.312231 (+ 222us) server_negotiation.cc:407] Connection header received
0929 17:58:19.312388 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:19.312391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:19.312442 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:19.312524 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:19.312811 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.313322 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.314663 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.315080 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.315201 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:19.315270 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:19.315457 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 17:58:19.315533 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 17:58:19.315689 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:19.311789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:42025 (local address 127.7.118.2:39705)
0929 17:58:19.312135 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:19.312147 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:19.312277 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:19.312534 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:19.312537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:19.312552 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:19.312700 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:19.312705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.313441 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.313446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:19.314533 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:19.314544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.315209 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.315212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:19.315313 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:19.315343 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 17:58:19.315440 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 17:58:19.315521 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":65,"thread_start_us":175,"threads_started":1}
I20260929 17:58:19.315699 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:19.311692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:45829 (local address 127.7.118.2:50187)
0929 17:58:19.312082 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:19.312093 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:19.312236 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:19.312517 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:19.312520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:19.312540 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:19.312697 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:19.312703 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.313446 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.313459 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:19.314519 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:19.314531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.315221 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.315225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:19.315349 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:19.315367 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 17:58:19.315464 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 17:58:19.315538 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":42,"spinlock_wait_cycles":21248,"thread_start_us":125,"threads_started":1}
I20260929 17:58:19.316630 15319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74fc73caa0cc464794bc4ad6f9df36cc" candidate_uuid: "22339151149f40f38aa45467baca4537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b73665a33caa4f5db919e1740a84f386" is_pre_election: true
I20260929 17:58:19.316663 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74fc73caa0cc464794bc4ad6f9df36cc" candidate_uuid: "22339151149f40f38aa45467baca4537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" is_pre_election: true
I20260929 17:58:19.316828 15319 raft_consensus.cc:2454] T 74fc73caa0cc464794bc4ad6f9df36cc P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22339151149f40f38aa45467baca4537 in term 0.
I20260929 17:58:19.316942 15599 raft_consensus.cc:2454] T 74fc73caa0cc464794bc4ad6f9df36cc P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22339151149f40f38aa45467baca4537 in term 0.
I20260929 17:58:19.317328 15393 leader_election.cc:302] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22339151149f40f38aa45467baca4537, b73665a33caa4f5db919e1740a84f386; no voters:
I20260929 17:58:19.317628 15681 raft_consensus.cc:2781] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:58:19.317698 15681 raft_consensus.cc:482] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:58:19.317735 15681 raft_consensus.cc:3037] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:19.318892 15681 raft_consensus.cc:504] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.319337 15681 leader_election.cc:288] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537 [CANDIDATE]: Term 1 election: Requested vote from peers b73665a33caa4f5db919e1740a84f386 (127.7.118.1:45829), 7a12a5e9fdae4cceabe756a3e18a2037 (127.7.118.3:42025)
I20260929 17:58:19.319772 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74fc73caa0cc464794bc4ad6f9df36cc" candidate_uuid: "22339151149f40f38aa45467baca4537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a12a5e9fdae4cceabe756a3e18a2037"
I20260929 17:58:19.319767 15319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74fc73caa0cc464794bc4ad6f9df36cc" candidate_uuid: "22339151149f40f38aa45467baca4537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b73665a33caa4f5db919e1740a84f386"
I20260929 17:58:19.319917 15599 raft_consensus.cc:3037] T 74fc73caa0cc464794bc4ad6f9df36cc P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:19.319947 15319 raft_consensus.cc:3037] T 74fc73caa0cc464794bc4ad6f9df36cc P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:19.321048 15319 raft_consensus.cc:2454] T 74fc73caa0cc464794bc4ad6f9df36cc P b73665a33caa4f5db919e1740a84f386 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22339151149f40f38aa45467baca4537 in term 1.
I20260929 17:58:19.321218 15599 raft_consensus.cc:2454] T 74fc73caa0cc464794bc4ad6f9df36cc P 7a12a5e9fdae4cceabe756a3e18a2037 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22339151149f40f38aa45467baca4537 in term 1.
I20260929 17:58:19.321431 15393 leader_election.cc:302] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22339151149f40f38aa45467baca4537, b73665a33caa4f5db919e1740a84f386; no voters:
I20260929 17:58:19.321638 15681 raft_consensus.cc:2781] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:58:19.321882 15681 raft_consensus.cc:686] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537 [term 1 LEADER]: Becoming Leader. State: Replica: 22339151149f40f38aa45467baca4537, State: Running, Role: LEADER
I20260929 17:58:19.322162 15681 consensus_queue.cc:237] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.323599 15714 tablet_replica.cc:439] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537: TxnStatusTablet state changed. Reason: New leader 22339151149f40f38aa45467baca4537. Latest consensus state: current_term: 1 leader_uuid: "22339151149f40f38aa45467baca4537" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } } }
I20260929 17:58:19.323737 15714 tablet_replica.cc:442] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537: This TxnStatusTablet replica's current role is: LEADER
I20260929 17:58:19.324047 15716 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 17:58:19.324234 15716 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 17:58:19.324995 15161 catalog_manager.cc:5495] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22339151149f40f38aa45467baca4537 (127.7.118.2). New cstate: current_term: 1 leader_uuid: "22339151149f40f38aa45467baca4537" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } health_report { overall_health: UNKNOWN } } }
I20260929 17:58:19.352447 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:19.347148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:42039 (local address 127.0.0.1:37690)
0929 17:58:19.347403 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:19.347422 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:19.347557 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:19.348004 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:19.348010 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:19.348027 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:58:19.348176 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:19.348182 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.349402 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.349406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:19.350365 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:19.350377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.351054 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.351057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:19.351131 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:19.351187 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:58:19.352130 (+ 943us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:58:19.352156 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 17:58:19.352228 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 17:58:19.352317 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260929 17:58:19.352514 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:19.347338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37690 (local address 127.7.118.2:42039)
0929 17:58:19.347495 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:19.347501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:19.347521 (+ 20us) server_negotiation.cc:407] Connection header received
0929 17:58:19.347716 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:19.347719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:19.347800 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:19.347915 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:58:19.348319 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.349270 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.350502 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.350912 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.351038 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:19.351340 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:58:19.351991 (+ 651us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:58:19.352126 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:19.352269 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 17:58:19.352341 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 17:58:19.356107 15319 raft_consensus.cc:1261] T 74fc73caa0cc464794bc4ad6f9df36cc P b73665a33caa4f5db919e1740a84f386 [term 1 FOLLOWER]: Refusing update from remote peer 22339151149f40f38aa45467baca4537: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:58:19.356374 15599 raft_consensus.cc:1261] T 74fc73caa0cc464794bc4ad6f9df36cc P 7a12a5e9fdae4cceabe756a3e18a2037 [term 1 FOLLOWER]: Refusing update from remote peer 22339151149f40f38aa45467baca4537: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:58:19.356662 15714 consensus_queue.cc:1035] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:58:19.356952 15681 consensus_queue.cc:1035] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:58:19.358857 15680 tablet_replica.cc:439] T 74fc73caa0cc464794bc4ad6f9df36cc P 7a12a5e9fdae4cceabe756a3e18a2037: TxnStatusTablet state changed. Reason: New leader 22339151149f40f38aa45467baca4537. Latest consensus state: current_term: 1 leader_uuid: "22339151149f40f38aa45467baca4537" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } } }
I20260929 17:58:19.359019 15680 tablet_replica.cc:442] T 74fc73caa0cc464794bc4ad6f9df36cc P 7a12a5e9fdae4cceabe756a3e18a2037: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:58:19.359139 15679 tablet_replica.cc:439] T 74fc73caa0cc464794bc4ad6f9df36cc P b73665a33caa4f5db919e1740a84f386: TxnStatusTablet state changed. Reason: New leader 22339151149f40f38aa45467baca4537. Latest consensus state: current_term: 1 leader_uuid: "22339151149f40f38aa45467baca4537" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } } }
I20260929 17:58:19.359267 15679 tablet_replica.cc:442] T 74fc73caa0cc464794bc4ad6f9df36cc P b73665a33caa4f5db919e1740a84f386: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:58:19.362087 15721 tablet_replica.cc:439] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "22339151149f40f38aa45467baca4537" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } } }
I20260929 17:58:19.362596 15679 tablet_replica.cc:439] T 74fc73caa0cc464794bc4ad6f9df36cc P b73665a33caa4f5db919e1740a84f386: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "22339151149f40f38aa45467baca4537" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } } }
I20260929 17:58:19.362706 15679 tablet_replica.cc:442] T 74fc73caa0cc464794bc4ad6f9df36cc P b73665a33caa4f5db919e1740a84f386: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:58:19.362341 15721 tablet_replica.cc:442] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537: This TxnStatusTablet replica's current role is: LEADER
I20260929 17:58:19.363353 15721 tablet_replica.cc:439] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "22339151149f40f38aa45467baca4537" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } } }
I20260929 17:58:19.363513 15721 tablet_replica.cc:442] T 74fc73caa0cc464794bc4ad6f9df36cc P 22339151149f40f38aa45467baca4537: This TxnStatusTablet replica's current role is: LEADER
I20260929 17:58:19.365483 15680 tablet_replica.cc:439] T 74fc73caa0cc464794bc4ad6f9df36cc P 7a12a5e9fdae4cceabe756a3e18a2037: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "22339151149f40f38aa45467baca4537" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } } }
I20260929 17:58:19.365624 15680 tablet_replica.cc:442] T 74fc73caa0cc464794bc4ad6f9df36cc P 7a12a5e9fdae4cceabe756a3e18a2037: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:58:19.379729 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:19.375590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:42039 (local address 127.7.118.3:38093)
0929 17:58:19.375937 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:19.375951 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:19.376044 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:19.376422 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:19.376424 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:19.376431 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:58:19.376521 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:19.376526 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.377563 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.377566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:19.378218 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:19.378227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.378884 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.378888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:19.378970 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:19.379039 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:58:19.379417 (+ 378us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:58:19.379425 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 17:58:19.379498 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 17:58:19.379585 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":120,"threads_started":1}
I20260929 17:58:19.379731 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:19.375719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:38093 (local address 127.7.118.2:42039)
0929 17:58:19.376171 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:19.376176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:19.376194 (+ 18us) server_negotiation.cc:407] Connection header received
0929 17:58:19.376248 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:19.376251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:19.376300 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:19.376397 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:58:19.376632 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.377429 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.378390 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.378758 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.378860 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:19.379180 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:58:19.379271 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:58:19.379352 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:19.379529 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 17:58:19.379589 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260929 17:58:19.387358 15319 raft_consensus.cc:1261] T 719f8716016b4924be10a4cb3b8e659e P b73665a33caa4f5db919e1740a84f386 [term 1 FOLLOWER]: Refusing update from remote peer 7a12a5e9fdae4cceabe756a3e18a2037: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:58:19.387408 15459 raft_consensus.cc:1261] T 719f8716016b4924be10a4cb3b8e659e P 22339151149f40f38aa45467baca4537 [term 1 FOLLOWER]: Refusing update from remote peer 7a12a5e9fdae4cceabe756a3e18a2037: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:58:19.387770 15680 consensus_queue.cc:1035] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:58:19.387955 15693 consensus_queue.cc:1035] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:58:19.390857 15686 mvcc.cc:205] Tried to move back new op lower bound from 7334726448686542848 to 7334726447984111616. Current Snapshot: MvccSnapshot[applied={T|T < 7334726448686542848}]
I20260929 17:58:19.395830 15687 mvcc.cc:205] Tried to move back new op lower bound from 7334726448686542848 to 7334726447984111616. Current Snapshot: MvccSnapshot[applied={T|T < 7334726448686542848}]
I20260929 17:58:19.411737 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:19.407432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:58883 (local address 127.7.118.3:42025)
0929 17:58:19.407571 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:19.407580 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:19.407771 (+ 191us) server_negotiation.cc:407] Connection header received
0929 17:58:19.407883 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:19.407885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:19.407940 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:19.408020 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:58:19.408354 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.409156 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.410521 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.410853 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.410947 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:19.411277 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:58:19.411352 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:58:19.411411 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:19.411520 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 17:58:19.411587 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 17:58:19.411804 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:19.407244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:42025 (local address 127.7.118.2:58883)
0929 17:58:19.407655 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:19.407671 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:19.407784 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:19.408057 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:19.408060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:19.408068 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:58:19.408201 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:19.408207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.409293 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.409296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:19.410357 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:19.410370 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.410957 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.410961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:19.411063 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:19.411131 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:58:19.411438 (+ 307us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:58:19.411445 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 17:58:19.411514 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 17:58:19.411604 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":118,"threads_started":1}
I20260929 17:58:19.437629 15162 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:58:19.437846 15162 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:58:19.440479 15162 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0fbcb6f395034e0c8288ed78a863662c: Sending DeleteTablet for 3 replicas of tablet 719f8716016b4924be10a4cb3b8e659e
I20260929 17:58:19.441262 15439 tablet_service.cc:1514] Processing DeleteTablet for tablet 719f8716016b4924be10a4cb3b8e659e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:58:19 UTC) from {username='oryx', principal='oryx/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:37688
I20260929 17:58:19.441278 15287 tablet_service.cc:1514] Processing DeleteTablet for tablet 719f8716016b4924be10a4cb3b8e659e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:58:19 UTC) from {username='oryx', principal='oryx/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:57706
I20260929 17:58:19.441305 15579 tablet_service.cc:1514] Processing DeleteTablet for tablet 719f8716016b4924be10a4cb3b8e659e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:58:19 UTC) from {username='oryx', principal='oryx/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:40890
I20260929 17:58:19.441489 15697 tablet_replica.cc:330] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037: stopping tablet replica
I20260929 17:58:19.441505 15698 tablet_replica.cc:330] T 719f8716016b4924be10a4cb3b8e659e P 22339151149f40f38aa45467baca4537: stopping tablet replica
I20260929 17:58:19.441609 15698 raft_consensus.cc:2229] T 719f8716016b4924be10a4cb3b8e659e P 22339151149f40f38aa45467baca4537 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:58:19.441609 15697 raft_consensus.cc:2229] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:58:19.441731 15698 raft_consensus.cc:2258] T 719f8716016b4924be10a4cb3b8e659e P 22339151149f40f38aa45467baca4537 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:58:19.441820 15697 raft_consensus.cc:2258] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:58:19.441913 15696 tablet_replica.cc:330] T 719f8716016b4924be10a4cb3b8e659e P b73665a33caa4f5db919e1740a84f386: stopping tablet replica
I20260929 17:58:19.442042 15696 raft_consensus.cc:2229] T 719f8716016b4924be10a4cb3b8e659e P b73665a33caa4f5db919e1740a84f386 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:58:19.442178 15696 raft_consensus.cc:2258] T 719f8716016b4924be10a4cb3b8e659e P b73665a33caa4f5db919e1740a84f386 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:58:19.445451 15696 ts_tablet_manager.cc:1905] T 719f8716016b4924be10a4cb3b8e659e P b73665a33caa4f5db919e1740a84f386: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:58:19.445443 15697 ts_tablet_manager.cc:1905] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:58:19.448551 15697 ts_tablet_manager.cc:1918] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:58:19.448551 15696 ts_tablet_manager.cc:1918] T 719f8716016b4924be10a4cb3b8e659e P b73665a33caa4f5db919e1740a84f386: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:58:19.448639 15696 log.cc:1196] T 719f8716016b4924be10a4cb3b8e659e P b73665a33caa4f5db919e1740a84f386: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-0/wal/wals/719f8716016b4924be10a4cb3b8e659e
I20260929 17:58:19.448660 15697 log.cc:1196] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-2/wal/wals/719f8716016b4924be10a4cb3b8e659e
I20260929 17:58:19.448905 15696 ts_tablet_manager.cc:1939] T 719f8716016b4924be10a4cb3b8e659e P b73665a33caa4f5db919e1740a84f386: Deleting consensus metadata
I20260929 17:58:19.448905 15697 ts_tablet_manager.cc:1939] T 719f8716016b4924be10a4cb3b8e659e P 7a12a5e9fdae4cceabe756a3e18a2037: Deleting consensus metadata
I20260929 17:58:19.450209 15149 catalog_manager.cc:4841] TS b73665a33caa4f5db919e1740a84f386 (127.7.118.1:45829): tablet 719f8716016b4924be10a4cb3b8e659e (table test-table [id=22ccf7c815f049f1a32d334ab5de39f8]) successfully deleted
I20260929 17:58:19.450291 15147 catalog_manager.cc:4841] TS 7a12a5e9fdae4cceabe756a3e18a2037 (127.7.118.3:42025): tablet 719f8716016b4924be10a4cb3b8e659e (table test-table [id=22ccf7c815f049f1a32d334ab5de39f8]) successfully deleted
I20260929 17:58:19.451155 15698 ts_tablet_manager.cc:1905] T 719f8716016b4924be10a4cb3b8e659e P 22339151149f40f38aa45467baca4537: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:58:19.454057 15698 ts_tablet_manager.cc:1918] T 719f8716016b4924be10a4cb3b8e659e P 22339151149f40f38aa45467baca4537: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:58:19.454165 15698 log.cc:1196] T 719f8716016b4924be10a4cb3b8e659e P 22339151149f40f38aa45467baca4537: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-1/wal/wals/719f8716016b4924be10a4cb3b8e659e
I20260929 17:58:19.454429 15698 ts_tablet_manager.cc:1939] T 719f8716016b4924be10a4cb3b8e659e P 22339151149f40f38aa45467baca4537: Deleting consensus metadata
I20260929 17:58:19.455360 15147 catalog_manager.cc:4841] TS 22339151149f40f38aa45467baca4537 (127.7.118.2:42039): tablet 719f8716016b4924be10a4cb3b8e659e (table test-table [id=22ccf7c815f049f1a32d334ab5de39f8]) successfully deleted
Sep 29 17:58:19 dist-test-slave-r7ph krb5kdc[15117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 17:58:19 dist-test-slave-r7ph krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:19 dist-test-slave-r7ph krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 17:58:19.475742 15211 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:19.468164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55370 (local address 127.7.118.62:46821)
0929 17:58:19.468309 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:19.468316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:19.468561 (+ 245us) server_negotiation.cc:407] Connection header received
0929 17:58:19.468693 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:19.468699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:19.468771 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:19.468885 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:19.469706 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.470704 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.471792 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.472157 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.472286 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:19.475638 (+ 3352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55370: BlockingRecv error: recv got EOF from 127.0.0.1:55370 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 17:58:19 dist-test-slave-r7ph krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM
I20260929 17:58:19.489684 15211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:19.477662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55382 (local address 127.7.118.62:46821)
0929 17:58:19.477950 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:19.477954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:19.478437 (+ 483us) server_negotiation.cc:407] Connection header received
0929 17:58:19.478574 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:19.478578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:19.478638 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:19.478738 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:19.479610 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.480611 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.481906 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.482282 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.482396 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:19.485659 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:19.485664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:19.485667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:19.485702 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:19.487635 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:19.488190 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:19.488194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:19.488196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:19.488304 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:19.488625 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:19.488628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:19.488630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:19.488913 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:19.489066 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:19.489346 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 17:58:19.489493 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203}
I20260929 17:58:19.492985 15162 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:58:19.493462 15162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:58:19.504230 15287 tablet_service.cc:1467] Processing CreateTablet for tablet 3b953814310f4a70af35de563dfcf306 (DEFAULT_TABLE table=test-table [id=684a553e4a3548b99018d4792bcef80f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:58:19.504577 15287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b953814310f4a70af35de563dfcf306. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:19.507072 15673 tablet_bootstrap.cc:492] T 3b953814310f4a70af35de563dfcf306 P b73665a33caa4f5db919e1740a84f386: Bootstrap starting.
I20260929 17:58:19.507532 15579 tablet_service.cc:1467] Processing CreateTablet for tablet 3b953814310f4a70af35de563dfcf306 (DEFAULT_TABLE table=test-table [id=684a553e4a3548b99018d4792bcef80f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:58:19.507838 15579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b953814310f4a70af35de563dfcf306. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:19.508106 15673 tablet_bootstrap.cc:654] T 3b953814310f4a70af35de563dfcf306 P b73665a33caa4f5db919e1740a84f386: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:19.509554 15673 tablet_bootstrap.cc:492] T 3b953814310f4a70af35de563dfcf306 P b73665a33caa4f5db919e1740a84f386: No bootstrap required, opened a new log
I20260929 17:58:19.509652 15673 ts_tablet_manager.cc:1397] T 3b953814310f4a70af35de563dfcf306 P b73665a33caa4f5db919e1740a84f386: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 17:58:19.510134 15673 raft_consensus.cc:348] T 3b953814310f4a70af35de563dfcf306 P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.510547 15673 raft_consensus.cc:374] T 3b953814310f4a70af35de563dfcf306 P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:19.510602 15673 raft_consensus.cc:729] T 3b953814310f4a70af35de563dfcf306 P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b73665a33caa4f5db919e1740a84f386, State: Initialized, Role: FOLLOWER
I20260929 17:58:19.510699 15439 tablet_service.cc:1467] Processing CreateTablet for tablet 3b953814310f4a70af35de563dfcf306 (DEFAULT_TABLE table=test-table [id=684a553e4a3548b99018d4792bcef80f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:58:19.510737 15673 consensus_queue.cc:260] T 3b953814310f4a70af35de563dfcf306 P b73665a33caa4f5db919e1740a84f386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.511018 15439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b953814310f4a70af35de563dfcf306. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:19.511376 15673 ts_tablet_manager.cc:1428] T 3b953814310f4a70af35de563dfcf306 P b73665a33caa4f5db919e1740a84f386: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260929 17:58:19.513509 15675 tablet_bootstrap.cc:492] T 3b953814310f4a70af35de563dfcf306 P 22339151149f40f38aa45467baca4537: Bootstrap starting.
I20260929 17:58:19.514137 15674 tablet_bootstrap.cc:492] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037: Bootstrap starting.
I20260929 17:58:19.514591 15675 tablet_bootstrap.cc:654] T 3b953814310f4a70af35de563dfcf306 P 22339151149f40f38aa45467baca4537: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:19.515087 15674 tablet_bootstrap.cc:654] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:19.516567 15674 tablet_bootstrap.cc:492] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037: No bootstrap required, opened a new log
I20260929 17:58:19.516675 15674 ts_tablet_manager.cc:1397] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 17:58:19.516813 15675 tablet_bootstrap.cc:492] T 3b953814310f4a70af35de563dfcf306 P 22339151149f40f38aa45467baca4537: No bootstrap required, opened a new log
I20260929 17:58:19.516939 15675 ts_tablet_manager.cc:1397] T 3b953814310f4a70af35de563dfcf306 P 22339151149f40f38aa45467baca4537: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 17:58:19.517107 15674 raft_consensus.cc:348] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.517237 15674 raft_consensus.cc:374] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:19.517268 15674 raft_consensus.cc:729] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a12a5e9fdae4cceabe756a3e18a2037, State: Initialized, Role: FOLLOWER
I20260929 17:58:19.517395 15674 consensus_queue.cc:260] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.517406 15675 raft_consensus.cc:348] T 3b953814310f4a70af35de563dfcf306 P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.517570 15675 raft_consensus.cc:374] T 3b953814310f4a70af35de563dfcf306 P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:19.517614 15675 raft_consensus.cc:729] T 3b953814310f4a70af35de563dfcf306 P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22339151149f40f38aa45467baca4537, State: Initialized, Role: FOLLOWER
I20260929 17:58:19.517656 15674 ts_tablet_manager.cc:1428] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:58:19.517758 15675 consensus_queue.cc:260] T 3b953814310f4a70af35de563dfcf306 P 22339151149f40f38aa45467baca4537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.518134 15675 ts_tablet_manager.cc:1428] T 3b953814310f4a70af35de563dfcf306 P 22339151149f40f38aa45467baca4537: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 17:58:19.752838 15693 raft_consensus.cc:482] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:58:19.752983 15693 raft_consensus.cc:504] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.753594 15693 leader_election.cc:288] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b73665a33caa4f5db919e1740a84f386 (127.7.118.1:45829), 22339151149f40f38aa45467baca4537 (127.7.118.2:42039)
I20260929 17:58:19.754086 15459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b953814310f4a70af35de563dfcf306" candidate_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22339151149f40f38aa45467baca4537" is_pre_election: true
I20260929 17:58:19.754086 15319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b953814310f4a70af35de563dfcf306" candidate_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b73665a33caa4f5db919e1740a84f386" is_pre_election: true
I20260929 17:58:19.754242 15459 raft_consensus.cc:2454] T 3b953814310f4a70af35de563dfcf306 P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a12a5e9fdae4cceabe756a3e18a2037 in term 0.
I20260929 17:58:19.754267 15319 raft_consensus.cc:2454] T 3b953814310f4a70af35de563dfcf306 P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a12a5e9fdae4cceabe756a3e18a2037 in term 0.
I20260929 17:58:19.754592 15533 leader_election.cc:302] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a12a5e9fdae4cceabe756a3e18a2037, b73665a33caa4f5db919e1740a84f386; no voters:
I20260929 17:58:19.754844 15693 raft_consensus.cc:2781] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:58:19.754921 15693 raft_consensus.cc:482] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:58:19.754978 15693 raft_consensus.cc:3037] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:19.756078 15693 raft_consensus.cc:504] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.756510 15693 leader_election.cc:288] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037 [CANDIDATE]: Term 1 election: Requested vote from peers b73665a33caa4f5db919e1740a84f386 (127.7.118.1:45829), 22339151149f40f38aa45467baca4537 (127.7.118.2:42039)
I20260929 17:58:19.756937 15459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b953814310f4a70af35de563dfcf306" candidate_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22339151149f40f38aa45467baca4537"
I20260929 17:58:19.756958 15319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b953814310f4a70af35de563dfcf306" candidate_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b73665a33caa4f5db919e1740a84f386"
I20260929 17:58:19.757076 15459 raft_consensus.cc:3037] T 3b953814310f4a70af35de563dfcf306 P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:19.757098 15319 raft_consensus.cc:3037] T 3b953814310f4a70af35de563dfcf306 P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:19.758311 15459 raft_consensus.cc:2454] T 3b953814310f4a70af35de563dfcf306 P 22339151149f40f38aa45467baca4537 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a12a5e9fdae4cceabe756a3e18a2037 in term 1.
I20260929 17:58:19.758529 15319 raft_consensus.cc:2454] T 3b953814310f4a70af35de563dfcf306 P b73665a33caa4f5db919e1740a84f386 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a12a5e9fdae4cceabe756a3e18a2037 in term 1.
I20260929 17:58:19.758693 15531 leader_election.cc:302] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22339151149f40f38aa45467baca4537, 7a12a5e9fdae4cceabe756a3e18a2037; no voters:
I20260929 17:58:19.758869 15693 raft_consensus.cc:2781] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:58:19.758983 15693 raft_consensus.cc:686] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 1 LEADER]: Becoming Leader. State: Replica: 7a12a5e9fdae4cceabe756a3e18a2037, State: Running, Role: LEADER
I20260929 17:58:19.759169 15693 consensus_queue.cc:237] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.761085 15162 catalog_manager.cc:5495] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a12a5e9fdae4cceabe756a3e18a2037 (127.7.118.3). New cstate: current_term: 1 leader_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } health_report { overall_health: HEALTHY } } }
I20260929 17:58:19.811189 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:19.807898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40904 (local address 127.7.118.3:42025)
0929 17:58:19.808063 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:19.808067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:19.808081 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:58:19.808127 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:19.808129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:19.808174 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:19.808285 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:58:19.808561 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.809099 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.809988 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.810223 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.810382 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:19.810661 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:58:19.810740 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:58:19.810820 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:19.810977 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 17:58:19.811036 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 17:58:19.813397 15319 raft_consensus.cc:1261] T 3b953814310f4a70af35de563dfcf306 P b73665a33caa4f5db919e1740a84f386 [term 1 FOLLOWER]: Refusing update from remote peer 7a12a5e9fdae4cceabe756a3e18a2037: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:58:19.813454 15459 raft_consensus.cc:1261] T 3b953814310f4a70af35de563dfcf306 P 22339151149f40f38aa45467baca4537 [term 1 FOLLOWER]: Refusing update from remote peer 7a12a5e9fdae4cceabe756a3e18a2037: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:58:19.813990 15693 consensus_queue.cc:1035] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:58:19.814162 15683 consensus_queue.cc:1035] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:58:19.821743 15162 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:58:19.821934 15162 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:58:19.824337 15162 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0fbcb6f395034e0c8288ed78a863662c: Sending DeleteTablet for 3 replicas of tablet 3b953814310f4a70af35de563dfcf306
I20260929 17:58:19.824985 15287 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b953814310f4a70af35de563dfcf306 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:58:19 UTC) from {username='oryx', principal='oryx/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:57706
I20260929 17:58:19.825049 15439 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b953814310f4a70af35de563dfcf306 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:58:19 UTC) from {username='oryx', principal='oryx/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:37688
I20260929 17:58:19.825202 15696 tablet_replica.cc:330] T 3b953814310f4a70af35de563dfcf306 P b73665a33caa4f5db919e1740a84f386: stopping tablet replica
I20260929 17:58:19.825227 15579 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b953814310f4a70af35de563dfcf306 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:58:19 UTC) from {username='oryx', principal='oryx/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:40890
I20260929 17:58:19.825332 15696 raft_consensus.cc:2229] T 3b953814310f4a70af35de563dfcf306 P b73665a33caa4f5db919e1740a84f386 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:58:19.825524 15696 raft_consensus.cc:2258] T 3b953814310f4a70af35de563dfcf306 P b73665a33caa4f5db919e1740a84f386 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:58:19.825721 15698 tablet_replica.cc:330] T 3b953814310f4a70af35de563dfcf306 P 22339151149f40f38aa45467baca4537: stopping tablet replica
I20260929 17:58:19.825882 15698 raft_consensus.cc:2229] T 3b953814310f4a70af35de563dfcf306 P 22339151149f40f38aa45467baca4537 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:58:19.826045 15698 raft_consensus.cc:2258] T 3b953814310f4a70af35de563dfcf306 P 22339151149f40f38aa45467baca4537 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:58:19.826129 15162 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 17:58:19.826372 15697 tablet_replica.cc:330] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037: stopping tablet replica
I20260929 17:58:19.826536 15697 raft_consensus.cc:2229] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 1 LEADER]: Raft consensus shutting down.
W20260929 17:58:19.826622 15162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:58:19.826761 15697 raft_consensus.cc:2258] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:58:19.826838 15696 ts_tablet_manager.cc:1905] T 3b953814310f4a70af35de563dfcf306 P b73665a33caa4f5db919e1740a84f386: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:58:19.827039 15698 ts_tablet_manager.cc:1905] T 3b953814310f4a70af35de563dfcf306 P 22339151149f40f38aa45467baca4537: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:58:19.829506 15698 ts_tablet_manager.cc:1918] T 3b953814310f4a70af35de563dfcf306 P 22339151149f40f38aa45467baca4537: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:58:19.829581 15698 log.cc:1196] T 3b953814310f4a70af35de563dfcf306 P 22339151149f40f38aa45467baca4537: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-1/wal/wals/3b953814310f4a70af35de563dfcf306
I20260929 17:58:19.829540 15696 ts_tablet_manager.cc:1918] T 3b953814310f4a70af35de563dfcf306 P b73665a33caa4f5db919e1740a84f386: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:58:19.829638 15696 log.cc:1196] T 3b953814310f4a70af35de563dfcf306 P b73665a33caa4f5db919e1740a84f386: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-0/wal/wals/3b953814310f4a70af35de563dfcf306
I20260929 17:58:19.829932 15698 ts_tablet_manager.cc:1939] T 3b953814310f4a70af35de563dfcf306 P 22339151149f40f38aa45467baca4537: Deleting consensus metadata
I20260929 17:58:19.829981 15696 ts_tablet_manager.cc:1939] T 3b953814310f4a70af35de563dfcf306 P b73665a33caa4f5db919e1740a84f386: Deleting consensus metadata
I20260929 17:58:19.830787 15147 catalog_manager.cc:4841] TS 22339151149f40f38aa45467baca4537 (127.7.118.2:42039): tablet 3b953814310f4a70af35de563dfcf306 (table test-table [id=684a553e4a3548b99018d4792bcef80f]) successfully deleted
I20260929 17:58:19.830858 15149 catalog_manager.cc:4841] TS b73665a33caa4f5db919e1740a84f386 (127.7.118.1:45829): tablet 3b953814310f4a70af35de563dfcf306 (table test-table [id=684a553e4a3548b99018d4792bcef80f]) successfully deleted
I20260929 17:58:19.832098 15439 tablet_service.cc:1467] Processing CreateTablet for tablet ecf8e610dd1747dabc6a9d8bfe414833 (DEFAULT_TABLE table=test-table [id=6e7657d6ae8c4aaf9c53b9ca3514b785]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:58:19.832360 15439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecf8e610dd1747dabc6a9d8bfe414833. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:19.834767 15675 tablet_bootstrap.cc:492] T ecf8e610dd1747dabc6a9d8bfe414833 P 22339151149f40f38aa45467baca4537: Bootstrap starting.
I20260929 17:58:19.835633 15675 tablet_bootstrap.cc:654] T ecf8e610dd1747dabc6a9d8bfe414833 P 22339151149f40f38aa45467baca4537: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:19.836624 15287 tablet_service.cc:1467] Processing CreateTablet for tablet ecf8e610dd1747dabc6a9d8bfe414833 (DEFAULT_TABLE table=test-table [id=6e7657d6ae8c4aaf9c53b9ca3514b785]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:58:19.836923 15287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecf8e610dd1747dabc6a9d8bfe414833. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:19.836927 15579 tablet_service.cc:1467] Processing CreateTablet for tablet ecf8e610dd1747dabc6a9d8bfe414833 (DEFAULT_TABLE table=test-table [id=6e7657d6ae8c4aaf9c53b9ca3514b785]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:58:19.837069 15675 tablet_bootstrap.cc:492] T ecf8e610dd1747dabc6a9d8bfe414833 P 22339151149f40f38aa45467baca4537: No bootstrap required, opened a new log
I20260929 17:58:19.837155 15675 ts_tablet_manager.cc:1397] T ecf8e610dd1747dabc6a9d8bfe414833 P 22339151149f40f38aa45467baca4537: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 17:58:19.837237 15579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecf8e610dd1747dabc6a9d8bfe414833. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:19.837548 15675 raft_consensus.cc:348] T ecf8e610dd1747dabc6a9d8bfe414833 P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.837647 15675 raft_consensus.cc:374] T ecf8e610dd1747dabc6a9d8bfe414833 P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:19.837674 15675 raft_consensus.cc:729] T ecf8e610dd1747dabc6a9d8bfe414833 P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22339151149f40f38aa45467baca4537, State: Initialized, Role: FOLLOWER
I20260929 17:58:19.837790 15675 consensus_queue.cc:260] T ecf8e610dd1747dabc6a9d8bfe414833 P 22339151149f40f38aa45467baca4537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.838016 15675 ts_tablet_manager.cc:1428] T ecf8e610dd1747dabc6a9d8bfe414833 P 22339151149f40f38aa45467baca4537: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:58:19.838785 15697 ts_tablet_manager.cc:1905] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:58:19.839588 15673 tablet_bootstrap.cc:492] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386: Bootstrap starting.
I20260929 17:58:19.840613 15673 tablet_bootstrap.cc:654] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:19.840948 15697 ts_tablet_manager.cc:1918] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:58:19.841013 15697 log.cc:1196] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-2/wal/wals/3b953814310f4a70af35de563dfcf306
I20260929 17:58:19.841259 15697 ts_tablet_manager.cc:1939] T 3b953814310f4a70af35de563dfcf306 P 7a12a5e9fdae4cceabe756a3e18a2037: Deleting consensus metadata
I20260929 17:58:19.841904 15673 tablet_bootstrap.cc:492] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386: No bootstrap required, opened a new log
I20260929 17:58:19.841928 15674 tablet_bootstrap.cc:492] T ecf8e610dd1747dabc6a9d8bfe414833 P 7a12a5e9fdae4cceabe756a3e18a2037: Bootstrap starting.
I20260929 17:58:19.841953 15147 catalog_manager.cc:4841] TS 7a12a5e9fdae4cceabe756a3e18a2037 (127.7.118.3:42025): tablet 3b953814310f4a70af35de563dfcf306 (table test-table [id=684a553e4a3548b99018d4792bcef80f]) successfully deleted
I20260929 17:58:19.842003 15673 ts_tablet_manager.cc:1397] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 17:58:19.842476 15673 raft_consensus.cc:348] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.842579 15673 raft_consensus.cc:374] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:19.842602 15673 raft_consensus.cc:729] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b73665a33caa4f5db919e1740a84f386, State: Initialized, Role: FOLLOWER
I20260929 17:58:19.842753 15674 tablet_bootstrap.cc:654] T ecf8e610dd1747dabc6a9d8bfe414833 P 7a12a5e9fdae4cceabe756a3e18a2037: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:19.842767 15673 consensus_queue.cc:260] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.843110 15673 ts_tablet_manager.cc:1428] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:58:19.850417 15674 tablet_bootstrap.cc:492] T ecf8e610dd1747dabc6a9d8bfe414833 P 7a12a5e9fdae4cceabe756a3e18a2037: No bootstrap required, opened a new log
I20260929 17:58:19.850567 15674 ts_tablet_manager.cc:1397] T ecf8e610dd1747dabc6a9d8bfe414833 P 7a12a5e9fdae4cceabe756a3e18a2037: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260929 17:58:19.851061 15674 raft_consensus.cc:348] T ecf8e610dd1747dabc6a9d8bfe414833 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.851191 15674 raft_consensus.cc:374] T ecf8e610dd1747dabc6a9d8bfe414833 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:19.851217 15674 raft_consensus.cc:729] T ecf8e610dd1747dabc6a9d8bfe414833 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a12a5e9fdae4cceabe756a3e18a2037, State: Initialized, Role: FOLLOWER
I20260929 17:58:19.851356 15674 consensus_queue.cc:260] T ecf8e610dd1747dabc6a9d8bfe414833 P 7a12a5e9fdae4cceabe756a3e18a2037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.851643 15674 ts_tablet_manager.cc:1428] T ecf8e610dd1747dabc6a9d8bfe414833 P 7a12a5e9fdae4cceabe756a3e18a2037: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:58:19.988833 15679 raft_consensus.cc:482] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:58:19.988984 15679 raft_consensus.cc:504] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.990120 15679 leader_election.cc:288] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22339151149f40f38aa45467baca4537 (127.7.118.2:42039), 7a12a5e9fdae4cceabe756a3e18a2037 (127.7.118.3:42025)
I20260929 17:58:19.994778 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:19.990577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:56075 (local address 127.7.118.2:42039)
0929 17:58:19.990856 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:19.990905 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:19.990962 (+ 57us) server_negotiation.cc:407] Connection header received
0929 17:58:19.991181 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:19.991186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:19.991245 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:19.991364 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:19.991682 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.992285 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.993573 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.994086 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.994231 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:19.994345 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:19.994551 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 17:58:19.994635 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20260929 17:58:19.994789 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:19.990497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:42039 (local address 127.7.118.1:56075)
0929 17:58:19.990837 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:19.990854 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:19.991031 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:19.991388 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:19.991391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:19.991411 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:19.991553 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:19.991560 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.992500 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.992506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:19.993398 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:19.993409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.994238 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.994242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:19.994382 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:19.994420 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 17:58:19.994529 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 17:58:19.994616 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":111,"threads_started":1}
I20260929 17:58:19.994964 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:19.991110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:42675 (local address 127.7.118.3:42025)
0929 17:58:19.991231 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:19.991237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:19.991325 (+ 88us) server_negotiation.cc:407] Connection header received
0929 17:58:19.991518 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:19.991521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:19.991580 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:19.991672 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:19.991944 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.992562 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.993769 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.994238 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.994427 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:19.994509 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:19.994734 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 17:58:19.994818 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 17:58:19.994982 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:19.990944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:42025 (local address 127.7.118.1:42675)
0929 17:58:19.991230 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:19.991245 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:19.991359 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:19.991683 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:19.991686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:19.991702 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:19.991833 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:19.991839 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.992711 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.992719 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:19.993624 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:19.993634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:19.994476 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:19.994480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:19.994600 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:19.994620 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 17:58:19.994718 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 17:58:19.994798 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":55,"threads_started":1}
I20260929 17:58:19.995309 15459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf8e610dd1747dabc6a9d8bfe414833" candidate_uuid: "b73665a33caa4f5db919e1740a84f386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22339151149f40f38aa45467baca4537" is_pre_election: true
I20260929 17:58:19.995481 15459 raft_consensus.cc:2454] T ecf8e610dd1747dabc6a9d8bfe414833 P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b73665a33caa4f5db919e1740a84f386 in term 0.
I20260929 17:58:19.995442 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf8e610dd1747dabc6a9d8bfe414833" candidate_uuid: "b73665a33caa4f5db919e1740a84f386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" is_pre_election: true
I20260929 17:58:19.995625 15599 raft_consensus.cc:2454] T ecf8e610dd1747dabc6a9d8bfe414833 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b73665a33caa4f5db919e1740a84f386 in term 0.
I20260929 17:58:19.996052 15251 leader_election.cc:302] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22339151149f40f38aa45467baca4537, b73665a33caa4f5db919e1740a84f386; no voters:
I20260929 17:58:19.996393 15679 raft_consensus.cc:2781] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:58:19.996474 15679 raft_consensus.cc:482] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:58:19.996518 15679 raft_consensus.cc:3037] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:19.997867 15679 raft_consensus.cc:504] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:19.998387 15679 leader_election.cc:288] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386 [CANDIDATE]: Term 1 election: Requested vote from peers 22339151149f40f38aa45467baca4537 (127.7.118.2:42039), 7a12a5e9fdae4cceabe756a3e18a2037 (127.7.118.3:42025)
I20260929 17:58:19.998903 15459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf8e610dd1747dabc6a9d8bfe414833" candidate_uuid: "b73665a33caa4f5db919e1740a84f386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22339151149f40f38aa45467baca4537"
I20260929 17:58:19.998934 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf8e610dd1747dabc6a9d8bfe414833" candidate_uuid: "b73665a33caa4f5db919e1740a84f386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a12a5e9fdae4cceabe756a3e18a2037"
I20260929 17:58:19.999033 15459 raft_consensus.cc:3037] T ecf8e610dd1747dabc6a9d8bfe414833 P 22339151149f40f38aa45467baca4537 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:19.999132 15599 raft_consensus.cc:3037] T ecf8e610dd1747dabc6a9d8bfe414833 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:20.000172 15459 raft_consensus.cc:2454] T ecf8e610dd1747dabc6a9d8bfe414833 P 22339151149f40f38aa45467baca4537 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b73665a33caa4f5db919e1740a84f386 in term 1.
I20260929 17:58:20.000490 15599 raft_consensus.cc:2454] T ecf8e610dd1747dabc6a9d8bfe414833 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b73665a33caa4f5db919e1740a84f386 in term 1.
I20260929 17:58:20.000676 15251 leader_election.cc:302] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22339151149f40f38aa45467baca4537, b73665a33caa4f5db919e1740a84f386; no voters:
I20260929 17:58:20.000917 15679 raft_consensus.cc:2781] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:58:20.001133 15679 raft_consensus.cc:686] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386 [term 1 LEADER]: Becoming Leader. State: Replica: b73665a33caa4f5db919e1740a84f386, State: Running, Role: LEADER
I20260929 17:58:20.001441 15679 consensus_queue.cc:237] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } }
I20260929 17:58:20.004226 15161 catalog_manager.cc:5495] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386 reported cstate change: term changed from 0 to 1, leader changed from <none> to b73665a33caa4f5db919e1740a84f386 (127.7.118.1). New cstate: current_term: 1 leader_uuid: "b73665a33caa4f5db919e1740a84f386" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73665a33caa4f5db919e1740a84f386" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 } health_report { overall_health: UNKNOWN } } }
I20260929 17:58:20.052542 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:20.047536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57712 (local address 127.7.118.1:45829)
0929 17:58:20.047930 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:20.047942 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:20.047960 (+ 18us) server_negotiation.cc:407] Connection header received
0929 17:58:20.048129 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:20.048133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:20.048241 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:20.048403 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:58:20.048806 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:20.049428 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:20.050498 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:20.050747 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:20.051165 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:20.051514 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:58:20.052039 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:58:20.052188 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:20.052270 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 17:58:20.052347 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":148,"threads_started":1}
I20260929 17:58:20.065538 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:20.061647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:42039 (local address 127.7.118.1:52645)
0929 17:58:20.062038 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:20.062052 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:20.062173 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:20.062485 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:20.062488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:20.062496 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:58:20.062633 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:20.062638 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:20.063453 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:20.063456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:20.064188 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:20.064199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:20.064678 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:20.064681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:20.064762 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:20.064817 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:58:20.065220 (+ 403us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:58:20.065227 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 17:58:20.065298 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 17:58:20.065375 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":107,"threads_started":1}
I20260929 17:58:20.065577 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:20.061840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:52645 (local address 127.7.118.2:42039)
0929 17:58:20.062017 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:20.062021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:20.062135 (+ 114us) server_negotiation.cc:407] Connection header received
0929 17:58:20.062290 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:20.062292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:20.062343 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:20.062413 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:58:20.062755 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:20.063316 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:20.064324 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:20.064546 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:20.064632 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:20.065017 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:58:20.065102 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:58:20.065171 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:20.065409 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 17:58:20.065469 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 17:58:20.072597 15599 raft_consensus.cc:1261] T ecf8e610dd1747dabc6a9d8bfe414833 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 1 FOLLOWER]: Refusing update from remote peer b73665a33caa4f5db919e1740a84f386: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:58:20.072728 15459 raft_consensus.cc:1261] T ecf8e610dd1747dabc6a9d8bfe414833 P 22339151149f40f38aa45467baca4537 [term 1 FOLLOWER]: Refusing update from remote peer b73665a33caa4f5db919e1740a84f386: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:58:20.073408 15755 consensus_queue.cc:1035] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a12a5e9fdae4cceabe756a3e18a2037" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:58:20.073696 15679 consensus_queue.cc:1035] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22339151149f40f38aa45467baca4537" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:58:20.095794 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:20.091405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:45829 (local address 127.7.118.2:49305)
0929 17:58:20.091772 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:20.091787 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:20.091979 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:20.092293 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:20.092313 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:20.092325 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:58:20.092470 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:20.092487 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:20.093550 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:20.093556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:20.094230 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:20.094240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:20.094795 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:20.094798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:20.094906 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:20.094946 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:58:20.095490 (+ 544us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:58:20.095500 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 17:58:20.095566 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 17:58:20.095662 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":196,"threads_started":1}
I20260929 17:58:20.095790 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:20.091444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:49305 (local address 127.7.118.1:45829)
0929 17:58:20.091592 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:20.091596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:20.091894 (+ 298us) server_negotiation.cc:407] Connection header received
0929 17:58:20.092114 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:20.092118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:20.092181 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:20.092264 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:58:20.092629 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:20.093404 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:20.094418 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:20.094660 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:20.094796 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:20.095280 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:58:20.095391 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:58:20.095481 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:20.095577 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 17:58:20.095638 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 17:58:20.129400 15162 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:58:20.129601 15162 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:58:20.132037 15162 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0fbcb6f395034e0c8288ed78a863662c: Sending DeleteTablet for 3 replicas of tablet ecf8e610dd1747dabc6a9d8bfe414833
I20260929 17:58:20.132743 15287 tablet_service.cc:1514] Processing DeleteTablet for tablet ecf8e610dd1747dabc6a9d8bfe414833 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:58:20 UTC) from {username='oryx', principal='oryx/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:57706
I20260929 17:58:20.132807 15579 tablet_service.cc:1514] Processing DeleteTablet for tablet ecf8e610dd1747dabc6a9d8bfe414833 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:58:20 UTC) from {username='oryx', principal='oryx/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:40890
I20260929 17:58:20.133021 15696 tablet_replica.cc:330] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386: stopping tablet replica
I20260929 17:58:20.133209 15696 raft_consensus.cc:2229] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:58:20.133302 15697 tablet_replica.cc:330] T ecf8e610dd1747dabc6a9d8bfe414833 P 7a12a5e9fdae4cceabe756a3e18a2037: stopping tablet replica
I20260929 17:58:20.133440 15697 raft_consensus.cc:2229] T ecf8e610dd1747dabc6a9d8bfe414833 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:58:20.133563 15697 raft_consensus.cc:2258] T ecf8e610dd1747dabc6a9d8bfe414833 P 7a12a5e9fdae4cceabe756a3e18a2037 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:58:20.133581 15696 raft_consensus.cc:2258] T ecf8e610dd1747dabc6a9d8bfe414833 P b73665a33caa4f5db919e1740a84f386 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:58:20.134066 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 15234
I20260929 17:58:20.134521 15697 ts_tablet_manager.cc:1905] T ecf8e610dd1747dabc6a9d8bfe414833 P 7a12a5e9fdae4cceabe756a3e18a2037: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:58:20.135098 15439 tablet_service.cc:1514] Processing DeleteTablet for tablet ecf8e610dd1747dabc6a9d8bfe414833 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:58:20 UTC) from {username='oryx', principal='oryx/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:37688
I20260929 17:58:20.135291 15698 tablet_replica.cc:330] T ecf8e610dd1747dabc6a9d8bfe414833 P 22339151149f40f38aa45467baca4537: stopping tablet replica
I20260929 17:58:20.135417 15698 raft_consensus.cc:2229] T ecf8e610dd1747dabc6a9d8bfe414833 P 22339151149f40f38aa45467baca4537 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:58:20.135540 15698 raft_consensus.cc:2258] T ecf8e610dd1747dabc6a9d8bfe414833 P 22339151149f40f38aa45467baca4537 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:58:20.137651 15697 ts_tablet_manager.cc:1918] T ecf8e610dd1747dabc6a9d8bfe414833 P 7a12a5e9fdae4cceabe756a3e18a2037: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:58:20.137703 15698 ts_tablet_manager.cc:1905] T ecf8e610dd1747dabc6a9d8bfe414833 P 22339151149f40f38aa45467baca4537: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:58:20.137758 15697 log.cc:1196] T ecf8e610dd1747dabc6a9d8bfe414833 P 7a12a5e9fdae4cceabe756a3e18a2037: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-2/wal/wals/ecf8e610dd1747dabc6a9d8bfe414833
I20260929 17:58:20.138029 15697 ts_tablet_manager.cc:1939] T ecf8e610dd1747dabc6a9d8bfe414833 P 7a12a5e9fdae4cceabe756a3e18a2037: Deleting consensus metadata
I20260929 17:58:20.139001 15147 catalog_manager.cc:4841] TS 7a12a5e9fdae4cceabe756a3e18a2037 (127.7.118.3:42025): tablet ecf8e610dd1747dabc6a9d8bfe414833 (table test-table [id=6e7657d6ae8c4aaf9c53b9ca3514b785]) successfully deleted
I20260929 17:58:20.140026 15698 ts_tablet_manager.cc:1918] T ecf8e610dd1747dabc6a9d8bfe414833 P 22339151149f40f38aa45467baca4537: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:58:20.140129 15698 log.cc:1196] T ecf8e610dd1747dabc6a9d8bfe414833 P 22339151149f40f38aa45467baca4537: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704657566865-7640-0/minicluster-data/ts-1/wal/wals/ecf8e610dd1747dabc6a9d8bfe414833
I20260929 17:58:20.140388 15698 ts_tablet_manager.cc:1939] T ecf8e610dd1747dabc6a9d8bfe414833 P 22339151149f40f38aa45467baca4537: Deleting consensus metadata
I20260929 17:58:20.141115 15147 catalog_manager.cc:4841] TS 22339151149f40f38aa45467baca4537 (127.7.118.2:42039): tablet ecf8e610dd1747dabc6a9d8bfe414833 (table test-table [id=6e7657d6ae8c4aaf9c53b9ca3514b785]) successfully deleted
I20260929 17:58:20.142858 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 15374
W20260929 17:58:20.142973 15149 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.118.1:45829 (error 108)
I20260929 17:58:20.144601 15771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:20.144050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:45829 (local address 127.0.0.1:57728)
0929 17:58:20.144346 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:20.144449 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.1:45829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":65,"threads_started":1}
W20260929 17:58:20.144781 15149 catalog_manager.cc:4584] TS b73665a33caa4f5db919e1740a84f386 (127.7.118.1:45829): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ecf8e610dd1747dabc6a9d8bfe414833: Network error: Client connection negotiation failed: client connection to 127.7.118.1:45829: connect: Connection refused (error 111)
I20260929 17:58:20.151098 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 15514
I20260929 17:58:20.160118 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 15130
2026-09-29T17:58:20Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3757 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:58:20 dist-test-slave-r7ph krb5kdc[15774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:58:20 dist-test-slave-r7ph krb5kdc[15774](info): set up 2 sockets
Sep 29 17:58:20 dist-test-slave-r7ph krb5kdc[15774](info): commencing operation
krb5kdc: starting...
W20260929 17:58:21.243969 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 17:58:21 dist-test-slave-r7ph krb5kdc[15774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:58:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:58:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 17:58:21.383333 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:40179
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:40817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:40179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:21.505034 15787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:21.505390 15787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:21.505497 15787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:21.510056 15787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:58:21.510166 15787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:21.510202 15787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:21.510272 15787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:58:21.510310 15787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:58:21.516438 15787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:40817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:40179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:40179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:21.517666 15787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:21.518886 15787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:21.528469 15793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:21.528466 15792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:21.528546 15787 server_base.cc:1034] running on GCE node
W20260929 17:58:21.528481 15795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:21.529417 15787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:21.530947 15787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:21.532157 15787 hybrid_clock.cc:648] HybridClock initialized: now 1790704701532131 us; error 47 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T17:58:21Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1379 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:58:21 dist-test-slave-r7ph krb5kdc[15800](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:58:21 dist-test-slave-r7ph krb5kdc[15800](info): set up 2 sockets
Sep 29 17:58:21 dist-test-slave-r7ph krb5kdc[15800](info): commencing operation
krb5kdc: starting...
W20260929 17:58:22.627419 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 17:58:22 dist-test-slave-r7ph krb5kdc[15800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:58:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:58:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 17:58:22.759143 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:42997
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:37339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:42997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:22.873981 15813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:22.874413 15813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:22.874464 15813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:22.878698 15813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:58:22.878782 15813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:22.878801 15813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:22.878818 15813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:58:22.878834 15813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:58:22.884663 15813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:37339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:42997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:42997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:22.885917 15813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:22.887075 15813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:58:22.897061 15813 server_base.cc:1034] running on GCE node
W20260929 17:58:22.896925 15819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:22.896932 15821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:22.896932 15818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:22.897974 15813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:22.899304 15813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:22.900508 15813 hybrid_clock.cc:648] HybridClock initialized: now 1790704702900491 us; error 34 us; skew 500 ppm
Sep 29 17:58:22 dist-test-slave-r7ph krb5kdc[15800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704702, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:22.903642 15813 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 17:58:22.904693 15813 webserver.cc:460] Webserver started at http://127.7.118.62:43519/ using document root <none> and password file <none>
I20260929 17:58:22.905316 15813 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:22.905440 15813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:22.905676 15813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:22.907946 15813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "e9b8da6c31e74b59b58802c89ea7f5b7"
format_stamp: "Formatted at 2026-09-29 17:58:22 on dist-test-slave-r7ph"
server_key: "b9984fdd76aab39c1935c85eb451bba3"
server_key_iv: "dd6be427a7bc194bb40898c7b775226c"
server_key_version: "encryptionkey@0"
I20260929 17:58:22.908548 15813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "e9b8da6c31e74b59b58802c89ea7f5b7"
format_stamp: "Formatted at 2026-09-29 17:58:22 on dist-test-slave-r7ph"
server_key: "b9984fdd76aab39c1935c85eb451bba3"
server_key_iv: "dd6be427a7bc194bb40898c7b775226c"
server_key_version: "encryptionkey@0"
I20260929 17:58:22.912719 15813 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 17:58:22.915575 15828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:22.917289 15813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:58:22.917488 15813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "e9b8da6c31e74b59b58802c89ea7f5b7"
format_stamp: "Formatted at 2026-09-29 17:58:22 on dist-test-slave-r7ph"
server_key: "b9984fdd76aab39c1935c85eb451bba3"
server_key_iv: "dd6be427a7bc194bb40898c7b775226c"
server_key_version: "encryptionkey@0"
I20260929 17:58:22.917637 15813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:22.935976 15813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:22.939818 15813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:22.940065 15813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:22.947667 15813 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:42997
I20260929 17:58:22.947721 15880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:42997 every 8 connection(s)
I20260929 17:58:22.948765 15813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 17:58:22.951977 15881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:22.956633 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 15813
I20260929 17:58:22.956805 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 17:58:22.957089 7640 external_mini_cluster.cc:1428] Setting key 93b265f75c8099b6331fe2749e7b9189
I20260929 17:58:22.958628 15881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7: Bootstrap starting.
I20260929 17:58:22.961186 15881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:22.962066 15881 log.cc:824] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:22.964443 15881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7: No bootstrap required, opened a new log
Sep 29 17:58:22 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:58:22.968652 15881 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b8da6c31e74b59b58802c89ea7f5b7" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42997 } }
I20260929 17:58:22.968921 15881 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:22.968997 15881 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b8da6c31e74b59b58802c89ea7f5b7, State: Initialized, Role: FOLLOWER
I20260929 17:58:22.969484 15881 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b8da6c31e74b59b58802c89ea7f5b7" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42997 } }
I20260929 17:58:22.969617 15881 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:58:22.969717 15881 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:58:22.969830 15881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:22.970906 15881 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b8da6c31e74b59b58802c89ea7f5b7" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42997 } }
I20260929 17:58:22.971290 15881 leader_election.cc:302] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9b8da6c31e74b59b58802c89ea7f5b7; no voters:
I20260929 17:58:22.971675 15881 leader_election.cc:288] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:58:22.971758 15886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:58:22.972117 15886 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [term 1 LEADER]: Becoming Leader. State: Replica: e9b8da6c31e74b59b58802c89ea7f5b7, State: Running, Role: LEADER
I20260929 17:58:22.972514 15886 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b8da6c31e74b59b58802c89ea7f5b7" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42997 } }
I20260929 17:58:22.973280 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:22.958731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42606 (local address 127.7.118.62:42997)
0929 17:58:22.959210 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:22.959226 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:22.959265 (+ 39us) server_negotiation.cc:407] Connection header received
0929 17:58:22.960039 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:22.960061 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:22.960440 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:22.960825 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:22.961670 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:22.962979 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:22.963920 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:22.964510 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:22.964690 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:22.967581 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:22.967586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:22.967603 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:22.967635 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:22.969799 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:22.970312 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:22.970319 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:22.970326 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:22.970465 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:22.970770 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:22.970780 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:22.970782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:22.971291 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:22.971510 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:22.971809 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 17:58:22.972324 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":160,"threads_started":1}
I20260929 17:58:22.973302 15881 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:58:22.974324 15887 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9b8da6c31e74b59b58802c89ea7f5b7. Latest consensus state: current_term: 1 leader_uuid: "e9b8da6c31e74b59b58802c89ea7f5b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b8da6c31e74b59b58802c89ea7f5b7" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42997 } } }
I20260929 17:58:22.974483 15887 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:22.974817 15888 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9b8da6c31e74b59b58802c89ea7f5b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b8da6c31e74b59b58802c89ea7f5b7" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42997 } } }
I20260929 17:58:22.974902 15888 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [sys.catalog]: This master's current role is: LEADER
W20260929 17:58:22.979286 15901 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:58:22.979403 15901 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:58:22.979494 15895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:58:22.980278 15895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:58:22.986543 15895 catalog_manager.cc:1347] Generated new cluster ID: 868555300962482ca013434984327138
I20260929 17:58:22.986641 15895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:58:22.993821 15895 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:58:22.995255 15895 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:58:23.007202 15895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7: Generated new TSK 0
I20260929 17:58:23.008191 15895 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.118.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
I20260929 17:58:23.042737 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:0
--local_ip_for_outbound_sockets=127.7.118.1
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:42997
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:37339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:23.170928 15907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:23.171235 15907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:23.171288 15907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:23.175935 15907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:23.176023 15907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:23.176107 15907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 17:58:23.182039 15907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:37339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:42997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:23.183341 15907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:23.184489 15907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:23.195276 15915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:23.195307 15912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:23.195353 15907 server_base.cc:1034] running on GCE node
W20260929 17:58:23.195307 15913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:23.196005 15907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:23.196785 15907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:23.197968 15907 hybrid_clock.cc:648] HybridClock initialized: now 1790704703197926 us; error 61 us; skew 500 ppm
Sep 29 17:58:23 dist-test-slave-r7ph krb5kdc[15800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704703, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:23.208652 15907 init.cc:377] Logged in from keytab as kudu/127.7.118.1@KRBTEST.COM (short username kudu)
I20260929 17:58:23.209787 15907 webserver.cc:460] Webserver started at http://127.7.118.1:45159/ using document root <none> and password file <none>
I20260929 17:58:23.210546 15907 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:23.210652 15907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:23.210915 15907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:23.213259 15907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/data/instance:
uuid: "003f99db58d448e4af8f9376e640da34"
format_stamp: "Formatted at 2026-09-29 17:58:23 on dist-test-slave-r7ph"
server_key: "c00455026d42ff51178f59b23b5fbf68"
server_key_iv: "a2cb7cdc771d3e1f529cd24f2b0bb600"
server_key_version: "encryptionkey@0"
I20260929 17:58:23.213908 15907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance:
uuid: "003f99db58d448e4af8f9376e640da34"
format_stamp: "Formatted at 2026-09-29 17:58:23 on dist-test-slave-r7ph"
server_key: "c00455026d42ff51178f59b23b5fbf68"
server_key_iv: "a2cb7cdc771d3e1f529cd24f2b0bb600"
server_key_version: "encryptionkey@0"
I20260929 17:58:23.218093 15907 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 17:58:23.221104 15922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:23.222990 15907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 17:58:23.223165 15907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "003f99db58d448e4af8f9376e640da34"
format_stamp: "Formatted at 2026-09-29 17:58:23 on dist-test-slave-r7ph"
server_key: "c00455026d42ff51178f59b23b5fbf68"
server_key_iv: "a2cb7cdc771d3e1f529cd24f2b0bb600"
server_key_version: "encryptionkey@0"
I20260929 17:58:23.223312 15907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:23.270411 15907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:23.275411 15907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:23.275676 15907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:23.276399 15907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:23.277487 15907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:23.277580 15907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:23.277659 15907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:23.277717 15907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:23.288499 15907 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:37733
I20260929 17:58:23.288523 16035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:37733 every 8 connection(s)
I20260929 17:58:23.289688 15907 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 17:58:23.291712 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 15907
I20260929 17:58:23.291901 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance
I20260929 17:58:23.292264 7640 external_mini_cluster.cc:1428] Setting key ea2e7f284768d57b3da5739811759542
WARNING: no policy specified for kudu/127.7.118.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.2@KRBTEST.COM" created.
Sep 29 17:58:23 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704703, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:58:23.315102 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:23.291960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:33473 (local address 127.7.118.62:42997)
0929 17:58:23.292125 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:23.292130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:23.294042 (+ 1912us) server_negotiation.cc:407] Connection header received
0929 17:58:23.295996 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:23.296003 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:23.296082 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:23.296194 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:23.298499 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:23.299166 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:23.301405 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:23.301912 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:23.302022 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:23.308907 (+ 6885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:23.308915 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:23.308921 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:23.308966 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:23.310858 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:23.312290 (+ 1432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:23.312295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:23.312296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:23.312416 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:23.313079 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:23.313084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:23.313085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:23.313370 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:23.313561 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:23.314668 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0929 17:58:23.314907 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 17:58:23.316200 16038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:23.292462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.1:33473)
0929 17:58:23.293753 (+ 1291us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:23.293821 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:23.295335 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:23.296513 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:23.296527 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:23.297338 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:23.298208 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:23.298236 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:23.299356 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:23.299363 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:23.301200 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:23.301219 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:23.302138 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:23.302144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:23.302451 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:23.303378 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:23.303403 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:23.308636 (+ 5233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:23.311218 (+ 2582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:23.311230 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:23.311255 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:23.312070 (+ 815us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:23.312547 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:23.312551 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:23.312553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:23.312833 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:23.313601 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:23.313614 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:23.314077 (+ 463us) client_negotiation.cc:769] Sending connection context
0929 17:58:23.314431 (+ 354us) client_negotiation.cc:240] Negotiation successful
0929 17:58:23.314898 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"thread_start_us":770,"threads_started":1}
I20260929 17:58:23.318324 16036 heartbeater.cc:347] Connected to a master server at 127.7.118.62:42997
I20260929 17:58:23.318674 16036 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:23.319626 16036 heartbeater.cc:511] Master 127.7.118.62:42997 requested a full tablet report, sending...
I20260929 17:58:23.322485 15845 ts_manager.cc:194] Registered new tserver with Master: 003f99db58d448e4af8f9376e640da34 (127.7.118.1:37733)
I20260929 17:58:23.324169 15845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.1@KRBTEST.COM'} at 127.7.118.1:33473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
I20260929 17:58:23.339629 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:0
--local_ip_for_outbound_sockets=127.7.118.2
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:42997
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:37339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:23.470748 16041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:23.471087 16041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:23.471134 16041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:23.477412 16041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:23.477502 16041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:23.477571 16041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 17:58:23.483703 16041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:37339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:42997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:23.484885 16041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:23.486387 16041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:23.500974 16047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:23.500968 16049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:23.501017 16046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:23.502138 16041 server_base.cc:1034] running on GCE node
I20260929 17:58:23.503031 16041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:23.504035 16041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:23.505244 16041 hybrid_clock.cc:648] HybridClock initialized: now 1790704703505200 us; error 61 us; skew 500 ppm
Sep 29 17:58:23 dist-test-slave-r7ph krb5kdc[15800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704703, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:23.508864 16041 init.cc:377] Logged in from keytab as kudu/127.7.118.2@KRBTEST.COM (short username kudu)
I20260929 17:58:23.511013 16041 webserver.cc:460] Webserver started at http://127.7.118.2:39643/ using document root <none> and password file <none>
I20260929 17:58:23.511713 16041 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:23.511844 16041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:23.512121 16041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:23.514820 16041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/data/instance:
uuid: "1c2c831360d74d549522ac32efc8beee"
format_stamp: "Formatted at 2026-09-29 17:58:23 on dist-test-slave-r7ph"
server_key: "a9ecbff8359f820ff301a419491a1303"
server_key_iv: "c1003c8435a693deb76e7a28f63cec55"
server_key_version: "encryptionkey@0"
I20260929 17:58:23.515539 16041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance:
uuid: "1c2c831360d74d549522ac32efc8beee"
format_stamp: "Formatted at 2026-09-29 17:58:23 on dist-test-slave-r7ph"
server_key: "a9ecbff8359f820ff301a419491a1303"
server_key_iv: "c1003c8435a693deb76e7a28f63cec55"
server_key_version: "encryptionkey@0"
I20260929 17:58:23.520416 16041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 17:58:23.523947 16056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:23.525848 16041 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:58:23.526067 16041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "1c2c831360d74d549522ac32efc8beee"
format_stamp: "Formatted at 2026-09-29 17:58:23 on dist-test-slave-r7ph"
server_key: "a9ecbff8359f820ff301a419491a1303"
server_key_iv: "c1003c8435a693deb76e7a28f63cec55"
server_key_version: "encryptionkey@0"
I20260929 17:58:23.526292 16041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:23.541518 16041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:23.545416 16041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:23.545668 16041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:23.546411 16041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:23.547334 16041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:23.547408 16041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:23.547482 16041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:23.547524 16041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:23.558130 16041 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:44125
I20260929 17:58:23.558301 16169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:44125 every 8 connection(s)
I20260929 17:58:23.559279 16041 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 17:58:23.568567 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 16041
I20260929 17:58:23.568701 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance
I20260929 17:58:23.569053 7640 external_mini_cluster.cc:1428] Setting key 83c695d21fb5a825d92b8e3363303929
Sep 29 17:58:23 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704703, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:58:23.577916 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:23.561424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:36655 (local address 127.7.118.62:42997)
0929 17:58:23.561604 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:23.561608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:23.563534 (+ 1926us) server_negotiation.cc:407] Connection header received
0929 17:58:23.564575 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:23.564579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:23.564633 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:23.564721 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:23.566342 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:23.567052 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:23.568985 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:23.569326 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:23.569415 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:23.572815 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:23.572821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:23.572826 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:23.572904 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:23.574908 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:23.575572 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:23.575577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:23.575581 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:23.575693 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:23.576106 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:23.576109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:23.576111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:23.576328 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:23.576454 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:23.577313 (+ 859us) server_negotiation.cc:299] Negotiation successful
0929 17:58:23.577660 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 17:58:23.579049 16172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:23.561898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.2:36655)
0929 17:58:23.563292 (+ 1394us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:23.563353 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:23.564311 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:23.564956 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:23.564964 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:23.565390 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:23.566106 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:23.566134 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:23.567205 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:23.567209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:23.568803 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:23.568818 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:23.569674 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:23.569677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:23.569863 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:23.570585 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:23.570610 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:23.572551 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:23.575083 (+ 2532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:23.575091 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:23.575117 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:23.575434 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:23.575807 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:23.575810 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:23.575813 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:23.575994 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:23.576456 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:23.576467 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:23.576899 (+ 432us) client_negotiation.cc:769] Sending connection context
0929 17:58:23.577168 (+ 269us) client_negotiation.cc:240] Negotiation successful
0929 17:58:23.577648 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1147,"thread_start_us":917,"threads_started":1}
WARNING: no policy specified for kudu/127.7.118.3@KRBTEST.COM; defaulting to no policy
I20260929 17:58:23.580678 16170 heartbeater.cc:347] Connected to a master server at 127.7.118.62:42997
I20260929 17:58:23.580997 16170 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:23.581571 16170 heartbeater.cc:511] Master 127.7.118.62:42997 requested a full tablet report, sending...
I20260929 17:58:23.583076 15845 ts_manager.cc:194] Registered new tserver with Master: 1c2c831360d74d549522ac32efc8beee (127.7.118.2:44125)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.3@KRBTEST.COM" created.
I20260929 17:58:23.583863 15845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.2@KRBTEST.COM'} at 127.7.118.2:36655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
I20260929 17:58:23.606950 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:0
--local_ip_for_outbound_sockets=127.7.118.3
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:42997
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:37339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:23.739987 16175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:23.740334 16175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:23.740383 16175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:23.745442 16175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:23.745720 16175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:23.745873 16175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 17:58:23.752027 16175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:37339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:42997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:23.753196 16175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:23.754439 16175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:23.766706 16180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:23.766702 16181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:23.767082 16183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:23.767297 16175 server_base.cc:1034] running on GCE node
I20260929 17:58:23.767844 16175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:23.768522 16175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:23.769778 16175 hybrid_clock.cc:648] HybridClock initialized: now 1790704703769762 us; error 42 us; skew 500 ppm
Sep 29 17:58:23 dist-test-slave-r7ph krb5kdc[15800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704703, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:23.773734 16175 init.cc:377] Logged in from keytab as kudu/127.7.118.3@KRBTEST.COM (short username kudu)
I20260929 17:58:23.775264 16175 webserver.cc:460] Webserver started at http://127.7.118.3:40435/ using document root <none> and password file <none>
I20260929 17:58:23.776090 16175 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:23.776484 16175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:23.776816 16175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:23.779651 16175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/data/instance:
uuid: "f5e8b464974a470a9e015bfff81c0e20"
format_stamp: "Formatted at 2026-09-29 17:58:23 on dist-test-slave-r7ph"
server_key: "124d43ae3f8a33e49e0525a66757dc44"
server_key_iv: "c7d1c44f0bf632be4b522e37aed0e697"
server_key_version: "encryptionkey@0"
I20260929 17:58:23.780362 16175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance:
uuid: "f5e8b464974a470a9e015bfff81c0e20"
format_stamp: "Formatted at 2026-09-29 17:58:23 on dist-test-slave-r7ph"
server_key: "124d43ae3f8a33e49e0525a66757dc44"
server_key_iv: "c7d1c44f0bf632be4b522e37aed0e697"
server_key_version: "encryptionkey@0"
I20260929 17:58:23.785427 16175 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 17:58:23.789055 16190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:23.790982 16175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 17:58:23.791199 16175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "f5e8b464974a470a9e015bfff81c0e20"
format_stamp: "Formatted at 2026-09-29 17:58:23 on dist-test-slave-r7ph"
server_key: "124d43ae3f8a33e49e0525a66757dc44"
server_key_iv: "c7d1c44f0bf632be4b522e37aed0e697"
server_key_version: "encryptionkey@0"
I20260929 17:58:23.791335 16175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:23.806209 16175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:23.810632 16175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:23.810883 16175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:23.811650 16175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:23.812708 16175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:23.812798 16175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:23.812984 16175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:23.813061 16175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:23.824952 16175 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:38031
I20260929 17:58:23.825063 16303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:38031 every 8 connection(s)
I20260929 17:58:23.825984 16175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 17:58:23.835419 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 16175
I20260929 17:58:23.835590 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance
I20260929 17:58:23.835876 7640 external_mini_cluster.cc:1428] Setting key 3867698415a019ceb42f0f8c4d7df66e
Sep 29 17:58:23 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704703, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:58:23.843323 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:23.830547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:36435 (local address 127.7.118.62:42997)
0929 17:58:23.830700 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:23.830704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:23.830719 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:58:23.831505 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:23.831508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:23.831591 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:23.831699 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:23.833233 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:23.833783 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:23.835317 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:23.835544 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:23.835693 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:23.838973 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:23.838976 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:23.838979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:23.839007 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:23.840632 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:23.841271 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:23.841280 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:23.841281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:23.841368 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:23.841757 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:23.841760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:23.841762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:23.841977 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:23.842088 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:23.842974 (+ 886us) server_negotiation.cc:299] Negotiation successful
0929 17:58:23.843134 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 17:58:23.844218 16306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:23.828775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.3:36435)
0929 17:58:23.830074 (+ 1299us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:23.830129 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:23.831225 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:23.831943 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:23.831951 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:23.832381 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:23.832953 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:23.832969 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:23.833958 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:23.833961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:23.835128 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:23.835153 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:23.835809 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:23.835816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:23.836041 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:23.836692 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:23.836710 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:23.838779 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:23.840801 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:23.840811 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:23.840832 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:23.841143 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:23.841470 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:23.841473 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:23.841475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:23.841647 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:23.842098 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:23.842108 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:23.842500 (+ 392us) client_negotiation.cc:769] Sending connection context
0929 17:58:23.842908 (+ 408us) client_negotiation.cc:240] Negotiation successful
0929 17:58:23.843305 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1070,"thread_start_us":689,"threads_started":1}
I20260929 17:58:23.845570 16304 heartbeater.cc:347] Connected to a master server at 127.7.118.62:42997
I20260929 17:58:23.845821 16304 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:23.846421 16304 heartbeater.cc:511] Master 127.7.118.62:42997 requested a full tablet report, sending...
I20260929 17:58:23.847602 15845 ts_manager.cc:194] Registered new tserver with Master: f5e8b464974a470a9e015bfff81c0e20 (127.7.118.3:38031)
I20260929 17:58:23.848351 15845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.3@KRBTEST.COM'} at 127.7.118.3:36435
I20260929 17:58:23.850952 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.7.118.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.118.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.118.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.1.keytab.
Entry for principal oryx/127.7.118.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.1.keytab.
WARNING: no policy specified for oryx/127.7.118.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.118.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.118.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.2.keytab.
Entry for principal oryx/127.7.118.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.2.keytab.
WARNING: no policy specified for oryx/127.7.118.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.118.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.118.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.3.keytab.
Entry for principal oryx/127.7.118.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.3.keytab.
I20260929 17:58:23.949895 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 15907
I20260929 17:58:23.955947 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 16041
I20260929 17:58:23.962868 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 16175
I20260929 17:58:23.968600 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 15813
I20260929 17:58:23.975383 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:42997
--webserver_interface=127.7.118.62
--webserver_port=43519
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:37339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:42997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:24.094888 16313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:24.095270 16313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:24.095350 16313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:24.099762 16313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:58:24.099875 16313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:24.099911 16313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:24.099956 16313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:58:24.099992 16313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:58:24.105957 16313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:37339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:42997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:42997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=43519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:24.107244 16313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:24.108392 16313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:24.118481 16321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:24.118577 16318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:24.118553 16319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:24.118839 16313 server_base.cc:1034] running on GCE node
I20260929 17:58:24.119545 16313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:24.120867 16313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:24.122063 16313 hybrid_clock.cc:648] HybridClock initialized: now 1790704704122011 us; error 76 us; skew 500 ppm
Sep 29 17:58:24 dist-test-slave-r7ph krb5kdc[15800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704704, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:24.125597 16313 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 17:58:24.126726 16313 webserver.cc:460] Webserver started at http://127.7.118.62:43519/ using document root <none> and password file <none>
I20260929 17:58:24.127377 16313 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:24.127501 16313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:24.133332 16313 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:58:24.136461 16328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:24.138236 16313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:58:24.138512 16313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "e9b8da6c31e74b59b58802c89ea7f5b7"
format_stamp: "Formatted at 2026-09-29 17:58:22 on dist-test-slave-r7ph"
server_key: "b9984fdd76aab39c1935c85eb451bba3"
server_key_iv: "dd6be427a7bc194bb40898c7b775226c"
server_key_version: "encryptionkey@0"
I20260929 17:58:24.139364 16313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:24.158010 16313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:24.163635 16313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:24.163889 16313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:24.172405 16313 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:42997
I20260929 17:58:24.172464 16380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:42997 every 8 connection(s)
I20260929 17:58:24.173872 16313 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 17:58:24.179451 16381 sys_catalog.cc:263] Verifying existing consensus state
I20260929 17:58:24.182034 16381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7: Bootstrap starting.
I20260929 17:58:24.183456 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 16313
I20260929 17:58:24.183947 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:37733
--local_ip_for_outbound_sockets=127.7.118.1
--tserver_master_addrs=127.7.118.62:42997
--webserver_port=45159
--webserver_interface=127.7.118.1
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:37339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.118.1
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:58:24.209380 16381 log.cc:824] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:24.218739 16381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:58:24.219197 16381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7: Bootstrap complete.
I20260929 17:58:24.223865 16381 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b8da6c31e74b59b58802c89ea7f5b7" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42997 } }
I20260929 17:58:24.224403 16381 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b8da6c31e74b59b58802c89ea7f5b7, State: Initialized, Role: FOLLOWER
I20260929 17:58:24.225030 16381 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b8da6c31e74b59b58802c89ea7f5b7" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42997 } }
I20260929 17:58:24.225207 16381 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:58:24.225296 16381 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:58:24.225425 16381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [term 1 FOLLOWER]: Advancing to term 2
I20260929 17:58:24.227783 16381 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b8da6c31e74b59b58802c89ea7f5b7" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42997 } }
I20260929 17:58:24.228188 16381 leader_election.cc:302] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9b8da6c31e74b59b58802c89ea7f5b7; no voters:
I20260929 17:58:24.228551 16381 leader_election.cc:288] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 17:58:24.228717 16385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 17:58:24.228981 16385 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [term 2 LEADER]: Becoming Leader. State: Replica: e9b8da6c31e74b59b58802c89ea7f5b7, State: Running, Role: LEADER
I20260929 17:58:24.229384 16385 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b8da6c31e74b59b58802c89ea7f5b7" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42997 } }
I20260929 17:58:24.230147 16381 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:58:24.231212 16386 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e9b8da6c31e74b59b58802c89ea7f5b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b8da6c31e74b59b58802c89ea7f5b7" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42997 } } }
I20260929 17:58:24.231386 16386 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:24.231674 16387 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9b8da6c31e74b59b58802c89ea7f5b7. Latest consensus state: current_term: 2 leader_uuid: "e9b8da6c31e74b59b58802c89ea7f5b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b8da6c31e74b59b58802c89ea7f5b7" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42997 } } }
I20260929 17:58:24.231757 16387 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7 [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:24.232517 16394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:58:24.236636 16394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:58:24.237383 16394 catalog_manager.cc:1259] Loaded cluster ID: 868555300962482ca013434984327138
I20260929 17:58:24.237458 16394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:58:24.240132 16394 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:58:24.241281 16394 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e9b8da6c31e74b59b58802c89ea7f5b7: Loaded TSK: 0
I20260929 17:58:24.243003 16394 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 17:58:24.324036 16383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:24.324334 16383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:24.324383 16383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:24.328812 16383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:24.328894 16383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:24.328956 16383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 17:58:24.335170 16383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:37339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.1.keytab
--principal=oryx/127.7.118.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:37733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=45159
--tserver_master_addrs=127.7.118.62:42997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:24.336396 16383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:24.337536 16383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:24.349939 16408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:24.349900 16410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:24.349956 16407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:24.350411 16383 server_base.cc:1034] running on GCE node
I20260929 17:58:24.350953 16383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:24.351752 16383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:24.353085 16383 hybrid_clock.cc:648] HybridClock initialized: now 1790704704353059 us; error 40 us; skew 500 ppm
Sep 29 17:58:24 dist-test-slave-r7ph krb5kdc[15800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704704, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:24.356735 16383 init.cc:377] Logged in from keytab as oryx/127.7.118.1@KRBTEST.COM (short username oryx)
I20260929 17:58:24.358052 16383 webserver.cc:460] Webserver started at http://127.7.118.1:45159/ using document root <none> and password file <none>
I20260929 17:58:24.358772 16383 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:24.358915 16383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:24.364313 16383 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:58:24.366870 16417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:24.368386 16383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:58:24.368539 16383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "003f99db58d448e4af8f9376e640da34"
format_stamp: "Formatted at 2026-09-29 17:58:23 on dist-test-slave-r7ph"
server_key: "c00455026d42ff51178f59b23b5fbf68"
server_key_iv: "a2cb7cdc771d3e1f529cd24f2b0bb600"
server_key_version: "encryptionkey@0"
I20260929 17:58:24.369174 16383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:24.384356 16383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:24.388376 16383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:24.388622 16383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:24.389410 16383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:24.390779 16383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:24.390839 16383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:24.390887 16383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:24.390933 16383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:24.401518 16383 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:37733
I20260929 17:58:24.401539 16530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:37733 every 8 connection(s)
I20260929 17:58:24.402776 16383 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 17:58:24.413156 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 16383
I20260929 17:58:24.413630 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:44125
--local_ip_for_outbound_sockets=127.7.118.2
--tserver_master_addrs=127.7.118.62:42997
--webserver_port=39643
--webserver_interface=127.7.118.2
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:37339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.118.2
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 17:58:24 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:24 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:24.417881 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:24.405273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.1:57177)
0929 17:58:24.406088 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:24.406151 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:24.407088 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:24.408318 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:24.408325 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:24.408749 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:24.409263 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:24.409279 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.410827 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.410830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:24.411829 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:24.411841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.412772 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.412776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:24.412985 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:24.413763 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:24.413792 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:24.416943 (+ 3151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":197,"threads_started":1}
W20260929 17:58:24.418602 16531 heartbeater.cc:644] Failed to heartbeat to 127.7.118.62:42997 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.118.62:42997: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
W20260929 17:58:24.418582 16533 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:24.405148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:57177 (local address 127.7.118.62:42997)
0929 17:58:24.405623 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:24.405640 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:24.406355 (+ 715us) server_negotiation.cc:407] Connection header received
0929 17:58:24.407531 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:24.407544 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:24.407891 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:24.408198 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:24.409492 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.410624 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.412008 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.412571 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.412718 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:24.417285 (+ 4567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.1:57177: BlockingRecv error: recv got EOF from 127.7.118.1:57177 (error 108)
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":143,"threads_started":1}
Sep 29 17:58:24 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:24 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:24.429100 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:24.419132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.1:58193)
0929 17:58:24.422491 (+ 3359us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:24.422514 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:24.422676 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:24.423019 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:24.423025 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:24.423262 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:24.423435 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:24.423448 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.424324 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.424330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:24.425416 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:24.425433 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.426009 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.426013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:24.426115 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:24.426643 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:24.426658 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:24.428930 (+ 2272us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3202}
W20260929 17:58:24.429275 16533 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:24.419244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:58193 (local address 127.7.118.62:42997)
0929 17:58:24.419400 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:24.419407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:24.422638 (+ 3231us) server_negotiation.cc:407] Connection header received
0929 17:58:24.422794 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:24.422796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:24.422856 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:24.422950 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:24.423615 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.424190 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.425600 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.425870 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.425980 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:24.429108 (+ 3128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.1:58193: BlockingRecv error: recv got EOF from 127.7.118.1:58193 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 17:58:24 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:24 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:24.436097 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:24.429871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.1:36503)
0929 17:58:24.430104 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:24.430118 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:24.430191 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:24.430536 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:24.430540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:24.430760 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:24.430855 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:24.430861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.432000 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.432003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:24.432556 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:24.432564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.433190 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.433193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:24.433261 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:24.433864 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:24.433882 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:24.435936 (+ 2054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150}
W20260929 17:58:24.436271 16533 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:24.430005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:36503 (local address 127.7.118.62:42997)
0929 17:58:24.430143 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:24.430148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:24.430164 (+ 16us) server_negotiation.cc:407] Connection header received
0929 17:58:24.430297 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:24.430301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:24.430354 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:24.430443 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:24.430972 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.431832 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.432681 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.433048 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.433160 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:24.436107 (+ 2947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.1:36503: BlockingRecv error: recv got EOF from 127.7.118.1:36503 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 17:58:24.436558 16531 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 17:58:24.541880 16535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:24.542243 16535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:24.542405 16535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:24.547375 16535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:24.547490 16535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:24.547600 16535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 17:58:24.553704 16535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:37339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.2.keytab
--principal=oryx/127.7.118.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:44125
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=39643
--tserver_master_addrs=127.7.118.62:42997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:24.555027 16535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:24.556149 16535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:24.567260 16541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:24.567250 16540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:24.567250 16543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:24.567654 16535 server_base.cc:1034] running on GCE node
I20260929 17:58:24.568163 16535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:24.568933 16535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:24.570240 16535 hybrid_clock.cc:648] HybridClock initialized: now 1790704704570204 us; error 67 us; skew 500 ppm
Sep 29 17:58:24 dist-test-slave-r7ph krb5kdc[15800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704704, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:24.575148 16535 init.cc:377] Logged in from keytab as oryx/127.7.118.2@KRBTEST.COM (short username oryx)
I20260929 17:58:24.576320 16535 webserver.cc:460] Webserver started at http://127.7.118.2:39643/ using document root <none> and password file <none>
I20260929 17:58:24.577009 16535 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:24.577102 16535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:24.581835 16535 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:58:24.584702 16550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:24.586481 16535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 17:58:24.586656 16535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "1c2c831360d74d549522ac32efc8beee"
format_stamp: "Formatted at 2026-09-29 17:58:23 on dist-test-slave-r7ph"
server_key: "a9ecbff8359f820ff301a419491a1303"
server_key_iv: "c1003c8435a693deb76e7a28f63cec55"
server_key_version: "encryptionkey@0"
I20260929 17:58:24.587327 16535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:24.613760 16535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:24.618983 16535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:24.619251 16535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:24.620042 16535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:24.621080 16535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:24.621161 16535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:24.621239 16535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:24.621289 16535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:24.633285 16535 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:44125
I20260929 17:58:24.633378 16663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:44125 every 8 connection(s)
I20260929 17:58:24.634610 16535 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 17:58:24.642616 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 16535
I20260929 17:58:24.643136 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:38031
--local_ip_for_outbound_sockets=127.7.118.3
--tserver_master_addrs=127.7.118.62:42997
--webserver_port=40435
--webserver_interface=127.7.118.3
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:37339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.118.3
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 17:58:24 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:24 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 17:58:24.648366 16533 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:24.637054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:44573 (local address 127.7.118.62:42997)
0929 17:58:24.637388 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:24.637393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:24.638463 (+ 1070us) server_negotiation.cc:407] Connection header received
0929 17:58:24.639943 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:24.639947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:24.640019 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:24.640131 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:24.641514 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.642065 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.643674 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.643904 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.644026 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:24.648229 (+ 4203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.2:44573: BlockingRecv error: recv got EOF from 127.7.118.2:44573 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
I20260929 17:58:24.649159 16666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:24.637525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.2:44573)
0929 17:58:24.638199 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:24.638320 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:24.639666 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:24.640350 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:24.640358 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:24.640787 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:24.641335 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:24.641351 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.642239 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.642294 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:24.643514 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:24.643528 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.644085 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.644089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:24.644333 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:24.645074 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:24.645092 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:24.648080 (+ 2988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":155,"threads_started":1}
W20260929 17:58:24.649900 16664 heartbeater.cc:644] Failed to heartbeat to 127.7.118.62:42997 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.118.62:42997: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Sep 29 17:58:24 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:24 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:24.658212 16666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:24.650559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.2:58347)
0929 17:58:24.650724 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:24.650741 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:24.650861 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:24.651375 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:24.651378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:24.651633 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:24.651744 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:24.651750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.653015 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.653019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:24.653985 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:24.653999 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.654873 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.654881 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:24.654989 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:24.655685 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:24.655699 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:24.658047 (+ 2348us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20260929 17:58:24.658418 16533 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:24.650745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:58347 (local address 127.7.118.62:42997)
0929 17:58:24.650994 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:24.651001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:24.651021 (+ 20us) server_negotiation.cc:407] Connection header received
0929 17:58:24.651097 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:24.651103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:24.651165 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:24.651276 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:24.651904 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.652838 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.654154 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.654648 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.654808 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:24.658273 (+ 3465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.2:58347: BlockingRecv error: recv got EOF from 127.7.118.2:58347 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
Sep 29 17:58:24 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:24 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:24.665635 16666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:24.659105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.2:37705)
0929 17:58:24.659292 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:24.659308 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:24.659422 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:24.659766 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:24.659768 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:24.659924 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:24.660016 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:24.660021 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.661166 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.661169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:24.661731 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:24.661741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.662514 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.662520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:24.662642 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:24.663334 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:24.663375 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:24.665515 (+ 2140us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 17:58:24.665944 16664 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 17:58:24.665992 16533 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:24.659298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:37705 (local address 127.7.118.62:42997)
0929 17:58:24.659437 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:24.659442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:24.659457 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:58:24.659509 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:24.659512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:24.659558 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:24.659643 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:24.660135 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.660968 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.661867 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.662305 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.662423 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:24.665715 (+ 3292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.2:37705: BlockingRecv error: recv got EOF from 127.7.118.2:37705 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 17:58:24.781533 16667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:24.782016 16667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:24.782207 16667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:24.787932 16667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:24.788038 16667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:24.788148 16667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 17:58:24.795625 16667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:37339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/krb5kdc/oryx_127.7.118.3.keytab
--principal=oryx/127.7.118.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:38031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=40435
--tserver_master_addrs=127.7.118.62:42997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:24.797456 16667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:24.799217 16667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:58:24.812191 16667 server_base.cc:1034] running on GCE node
W20260929 17:58:24.812289 16672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:24.812256 16673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:24.812433 16675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:24.812989 16667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:24.813795 16667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:24.815089 16667 hybrid_clock.cc:648] HybridClock initialized: now 1790704704815034 us; error 81 us; skew 500 ppm
Sep 29 17:58:24 dist-test-slave-r7ph krb5kdc[15800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704704, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.118.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:24.819202 16667 init.cc:377] Logged in from keytab as oryx/127.7.118.3@KRBTEST.COM (short username oryx)
I20260929 17:58:24.820359 16667 webserver.cc:460] Webserver started at http://127.7.118.3:40435/ using document root <none> and password file <none>
I20260929 17:58:24.821139 16667 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:24.821238 16667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:24.826709 16667 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 17:58:24.829668 16682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:24.831708 16667 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 17:58:24.831866 16667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "f5e8b464974a470a9e015bfff81c0e20"
format_stamp: "Formatted at 2026-09-29 17:58:23 on dist-test-slave-r7ph"
server_key: "124d43ae3f8a33e49e0525a66757dc44"
server_key_iv: "c7d1c44f0bf632be4b522e37aed0e697"
server_key_version: "encryptionkey@0"
I20260929 17:58:24.832585 16667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:24.845249 16667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:24.849231 16667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:24.849426 16667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:24.850082 16667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:24.851024 16667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:24.851073 16667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:24.851116 16667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:24.851132 16667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:24.865960 16667 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:38031
I20260929 17:58:24.866098 16795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:38031 every 8 connection(s)
I20260929 17:58:24.867239 16667 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 17:58:24.871862 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 16667
Sep 29 17:58:24 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:24 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 17:58:24.883009 16533 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:24.869419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:60915 (local address 127.7.118.62:42997)
0929 17:58:24.869684 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:24.869766 (+ 82us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:24.871229 (+ 1463us) server_negotiation.cc:407] Connection header received
0929 17:58:24.872338 (+ 1109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:24.872343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:24.872407 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:24.872494 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:24.874782 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.875941 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.878053 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.878523 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.878636 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:24.882859 (+ 4223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.3:60915: BlockingRecv error: recv got EOF from 127.7.118.3:60915 (error 108)
Metrics: {"mutex_wait_us":134,"server-negotiator.queue_time_us":117}
I20260929 17:58:24.884145 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:24.870096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.3:60915)
0929 17:58:24.871045 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:24.871095 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:24.871977 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:24.873017 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:24.873029 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:24.873674 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:24.874614 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:24.874643 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.876288 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.876292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:24.877872 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:24.877886 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.878717 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.878722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:24.878992 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:24.879873 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:24.879892 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:24.882737 (+ 2845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":158,"threads_started":1}
W20260929 17:58:24.884964 16796 heartbeater.cc:644] Failed to heartbeat to 127.7.118.62:42997 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.118.62:42997: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
I20260929 17:58:24.885947 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:24.873181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42608 (local address 127.7.118.62:42997)
0929 17:58:24.873428 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:24.873431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:24.873445 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:58:24.873579 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:24.873581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:24.873620 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:24.873710 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:24.874765 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.875941 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.877622 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.878169 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.878424 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:24.880672 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:24.880680 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:24.880699 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:24.880734 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:24.882957 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:24.883671 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:24.883678 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:24.883684 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:24.883870 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:24.884236 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:24.884239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:24.884241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:24.884858 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:24.885097 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:24.885421 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 17:58:24.885729 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":172,"thread_start_us":132,"threads_started":1}
Sep 29 17:58:24 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:24 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:24.891978 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:24.885528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.3:39323)
0929 17:58:24.885716 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:24.885731 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:24.885816 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:24.886181 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:24.886184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:24.886398 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:24.886502 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:24.886508 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.887487 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.887491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:24.888282 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:24.888294 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.888862 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.888866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:24.889108 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:24.889669 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:24.889681 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:24.891814 (+ 2133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116}
W20260929 17:58:24.892367 16533 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:24.885621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:39323 (local address 127.7.118.62:42997)
0929 17:58:24.885767 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:24.885771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:24.885785 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:58:24.885923 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:24.885926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:24.885967 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:24.886055 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:24.886611 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.887355 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.888410 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.888725 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.888827 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:24.892190 (+ 3363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.3:39323: BlockingRecv error: recv got EOF from 127.7.118.3:39323 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 17:58:24 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:24 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:24.899142 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:24.893074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.3:42617)
0929 17:58:24.893312 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:24.893327 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:24.893425 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:24.894031 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:24.894034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:24.894191 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:24.894362 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:24.894369 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.895296 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.895299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:24.895865 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:24.895875 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.896426 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.896429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:24.896509 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:24.897099 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:24.897112 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:24.899024 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97}
W20260929 17:58:24.899417 16796 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 17:58:24.899866 16533 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:24.893385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:42617 (local address 127.7.118.62:42997)
0929 17:58:24.893556 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:24.893559 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:24.893573 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:58:24.893622 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:24.893625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:24.893665 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:24.893932 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:24.894536 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.895137 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.896004 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:24.896242 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:24.896373 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:24.899748 (+ 3375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.3:42617: BlockingRecv error: recv got EOF from 127.7.118.3:42617 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 29 17:58:25 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:25 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:25.443360 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:25.437333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.1:40543)
0929 17:58:25.437657 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:25.437671 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:25.437780 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:25.438084 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:25.438086 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:25.438335 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:25.438465 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:25.438472 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:25.439337 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:25.439342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:25.439987 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:25.439997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:25.440510 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:25.440514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:25.440613 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:25.441192 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:25.441204 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:25.443167 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":117,"threads_started":1}
W20260929 17:58:25.443670 16802 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:25.437523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:40543 (local address 127.7.118.62:42997)
0929 17:58:25.437771 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:25.437775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:25.437792 (+ 17us) server_negotiation.cc:407] Connection header received
0929 17:58:25.437904 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:25.437907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:25.437957 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:25.438040 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:25.438639 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:25.439214 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:25.440118 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:25.440388 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:25.440504 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:25.443315 (+ 2811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.1:40543: BlockingRecv error: recv got EOF from 127.7.118.1:40543 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":92,"threads_started":1}
Sep 29 17:58:25 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:25 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:25.673321 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:25.666857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.2:53859)
0929 17:58:25.667151 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:25.667165 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:25.667273 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:25.667584 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:25.667587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:25.667823 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:25.667953 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:25.667960 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:25.668862 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:25.668865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:25.669453 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:25.669463 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:25.670040 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:25.670044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:25.670117 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:25.670835 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:25.670848 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:25.673102 (+ 2254us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":107,"threads_started":1}
W20260929 17:58:25.673395 16802 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:25.667004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:53859 (local address 127.7.118.62:42997)
0929 17:58:25.667174 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:25.667178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:25.667300 (+ 122us) server_negotiation.cc:407] Connection header received
0929 17:58:25.667356 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:25.667358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:25.667408 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:25.667495 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:25.668106 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:25.668668 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:25.669608 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:25.669881 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:25.669994 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:25.673272 (+ 3278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.2:53859: BlockingRecv error: recv got EOF from 127.7.118.2:53859 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 29 17:58:25 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:25 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:25.906594 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:25.900144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.3:49799)
0929 17:58:25.900472 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:25.900492 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:25.900617 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:25.900939 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:25.900942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:25.901145 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:25.901249 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:25.901254 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:25.902052 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:25.902055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:25.902838 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:25.902849 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:25.903352 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:25.903356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:25.903428 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:25.904112 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:25.904126 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:25.906440 (+ 2314us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":81,"threads_started":1}
W20260929 17:58:25.906744 16802 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:25.900212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:49799 (local address 127.7.118.62:42997)
0929 17:58:25.900375 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:25.900380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:25.900576 (+ 196us) server_negotiation.cc:407] Connection header received
0929 17:58:25.900727 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:25.900731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:25.900787 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:25.900879 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:25.901389 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:25.901934 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:25.902972 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:25.903204 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:25.903291 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:25.906620 (+ 3329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.3:49799: BlockingRecv error: recv got EOF from 127.7.118.3:49799 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 17:58:26 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:26 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:26.450760 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:26.444423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.1:36051)
0929 17:58:26.444752 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:26.444768 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:26.444896 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:26.445215 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:26.445218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:26.445457 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:26.445583 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:26.445591 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:26.446449 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:26.446453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:26.447122 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:26.447132 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:26.447613 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:26.447616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:26.447685 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:26.448377 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:26.448391 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:26.450571 (+ 2180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":99,"threads_started":1}
W20260929 17:58:26.450847 16806 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:26.444450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:36051 (local address 127.7.118.62:42997)
0929 17:58:26.444756 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:26.444760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:26.444867 (+ 107us) server_negotiation.cc:407] Connection header received
0929 17:58:26.444999 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:26.445005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:26.445075 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:26.445160 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:26.445714 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:26.446346 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:26.447261 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:26.447503 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:26.447591 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:26.450718 (+ 3127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.1:36051: BlockingRecv error: recv got EOF from 127.7.118.1:36051 (error 108)
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":96,"threads_started":1}
Sep 29 17:58:26 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:26 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:26.680609 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:26.674392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.2:37219)
0929 17:58:26.674675 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:26.674695 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:26.674807 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:26.675056 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:26.675058 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:26.675237 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:26.675368 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:26.675388 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:26.676543 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:26.676549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:26.677345 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:26.677358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:26.677859 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:26.677862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:26.677928 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:26.678475 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:26.678487 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:26.680465 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":109,"threads_started":1}
W20260929 17:58:26.680821 16806 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:26.674495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:37219 (local address 127.7.118.62:42997)
0929 17:58:26.674640 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:26.674644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:26.674793 (+ 149us) server_negotiation.cc:407] Connection header received
0929 17:58:26.674882 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:26.674884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:26.674933 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:26.675018 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:26.675553 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:26.676409 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:26.677479 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:26.677735 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:26.677821 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:26.680634 (+ 2813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.2:37219: BlockingRecv error: recv got EOF from 127.7.118.2:37219 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 17:58:26 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:26 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:26.921149 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:26.907504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.3:50533)
0929 17:58:26.907827 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:26.907842 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:26.907948 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:26.908179 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:26.908181 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:26.908358 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:26.908455 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:26.908461 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:26.909175 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:26.909177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:26.909741 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:26.909748 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:26.910183 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:26.910186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:26.910325 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:26.918473 (+ 8148us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:26.918493 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:26.921001 (+ 2508us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":90,"threads_started":1}
W20260929 17:58:26.921298 16806 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:26.907577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:50533 (local address 127.7.118.62:42997)
0929 17:58:26.907723 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:26.907730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:26.907931 (+ 201us) server_negotiation.cc:407] Connection header received
0929 17:58:26.908025 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:26.908028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:26.908075 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:26.908142 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:26.908559 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:26.909080 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:26.909862 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:26.910078 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:26.910161 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:26.921165 (+ 11004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.3:50533: BlockingRecv error: recv got EOF from 127.7.118.3:50533 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 17:58:27 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:27 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:27.458491 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:27.451812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.1:36013)
0929 17:58:27.452291 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:27.452308 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:27.452436 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:27.452789 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:27.452792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:27.453014 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:27.453164 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:27.453178 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:27.454082 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:27.454088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:27.454770 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:27.454780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:27.455356 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:27.455361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:27.455451 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:27.456098 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:27.456110 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:27.458364 (+ 2254us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":89,"threads_started":1}
W20260929 17:58:27.458776 16810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:27.451948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:36013 (local address 127.7.118.62:42997)
0929 17:58:27.452244 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:27.452252 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:27.452394 (+ 142us) server_negotiation.cc:407] Connection header received
0929 17:58:27.452556 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:27.452560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:27.452627 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:27.452755 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:27.453338 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:27.453952 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:27.454908 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:27.455212 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:27.455362 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:27.458565 (+ 3203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.1:36013: BlockingRecv error: recv got EOF from 127.7.118.1:36013 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":59,"threads_started":1}
Sep 29 17:58:27 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:27 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:27.687968 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:27.681554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.2:50147)
0929 17:58:27.681845 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:27.681863 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:27.681979 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:27.682338 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:27.682341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:27.682524 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:27.682625 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:27.682630 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:27.683477 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:27.683480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:27.684091 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:27.684101 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:27.684595 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:27.684598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:27.684665 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:27.685359 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:27.685373 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:27.687759 (+ 2386us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":117,"threads_started":1}
W20260929 17:58:27.688057 16810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:27.681696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:50147 (local address 127.7.118.62:42997)
0929 17:58:27.681849 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:27.681853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:27.681947 (+ 94us) server_negotiation.cc:407] Connection header received
0929 17:58:27.682063 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:27.682065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:27.682119 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:27.682208 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:27.682772 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:27.683349 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:27.684244 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:27.684465 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:27.684559 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:27.687933 (+ 3374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.2:50147: BlockingRecv error: recv got EOF from 127.7.118.2:50147 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 17:58:27 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:27 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:27.928339 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:27.922139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.3:46737)
0929 17:58:27.922606 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:27.922624 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:27.922800 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:27.923092 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:27.923095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:27.923339 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:27.923433 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:27.923439 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:27.924199 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:27.924203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:27.924771 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:27.924779 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:27.925241 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:27.925244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:27.925330 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:27.925914 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:27.925926 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:27.928113 (+ 2187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":103,"threads_started":1}
W20260929 17:58:27.928380 16810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:27.922235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:46737 (local address 127.7.118.62:42997)
0929 17:58:27.922408 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:27.922411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:27.922740 (+ 329us) server_negotiation.cc:407] Connection header received
0929 17:58:27.922917 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:27.922921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:27.922979 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:27.923070 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:27.923552 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:27.924079 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:27.924892 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:27.925120 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:27.925237 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:27.928272 (+ 3035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.3:46737: BlockingRecv error: recv got EOF from 127.7.118.3:46737 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 17:58:28 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:28 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:28.471498 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:28.459335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.1:46221)
0929 17:58:28.459686 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:28.459701 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:28.459811 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:28.460236 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:28.460238 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:28.460431 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:28.460525 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:28.460531 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:28.461315 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:28.461318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:28.461922 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:28.461938 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:28.462532 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:28.462536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:28.462630 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:28.468896 (+ 6266us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:28.468911 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:28.471311 (+ 2400us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":93,"threads_started":1}
W20260929 17:58:28.471608 16814 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:28.459463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:46221 (local address 127.7.118.62:42997)
0929 17:58:28.459727 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:28.459731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:28.459764 (+ 33us) server_negotiation.cc:407] Connection header received
0929 17:58:28.460043 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:28.460046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:28.460096 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:28.460184 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:28.460635 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:28.461186 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:28.462065 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:28.462392 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:28.462518 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:28.471470 (+ 8952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.1:46221: BlockingRecv error: recv got EOF from 127.7.118.1:46221 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":72,"threads_started":1}
Sep 29 17:58:28 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:28 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:28.695946 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:28.689044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.2:35471)
0929 17:58:28.689381 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:28.689399 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:28.689536 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:28.689842 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:28.689845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:28.690075 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:28.690205 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:28.690213 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:28.691224 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:28.691228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:28.691987 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:28.691999 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:28.692593 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:28.692617 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:28.692698 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:28.693410 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:28.693424 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:28.695816 (+ 2392us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":109,"threads_started":1}
W20260929 17:58:28.696213 16814 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:28.689179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:35471 (local address 127.7.118.62:42997)
0929 17:58:28.689357 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:28.689362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:28.689495 (+ 133us) server_negotiation.cc:407] Connection header received
0929 17:58:28.689664 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:28.689668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:28.689723 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:28.689814 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:28.690422 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:28.690968 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:28.692145 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:28.692394 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:28.692500 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:28.696039 (+ 3539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.2:35471: BlockingRecv error: recv got EOF from 127.7.118.2:35471 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 17:58:28 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:28 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:28.935743 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:28.929226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.3:37149)
0929 17:58:28.929640 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:28.929658 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:28.929793 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:28.930209 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:28.930212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:28.930444 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:28.930552 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:28.930558 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:28.931440 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:28.931444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:28.932087 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:28.932097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:28.932604 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:28.932607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:28.932698 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:28.933296 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:28.933308 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:28.935517 (+ 2209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":141,"threads_started":1}
W20260929 17:58:28.935832 16814 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:28.929337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:37149 (local address 127.7.118.62:42997)
0929 17:58:28.929531 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:28.929537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:28.929733 (+ 196us) server_negotiation.cc:407] Connection header received
0929 17:58:28.929909 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:28.929912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:28.929968 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:28.930156 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:28.930679 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:28.931310 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:28.932211 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:28.932443 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:28.932579 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:28.935694 (+ 3115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.3:37149: BlockingRecv error: recv got EOF from 127.7.118.3:37149 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 29 17:58:29 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:29 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:29.478286 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:29.472459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.1:44321)
0929 17:58:29.472750 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:29.472767 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:29.472896 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:29.473185 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:29.473187 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:29.473374 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:29.473483 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:29.473490 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:29.474409 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:29.474412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:29.474990 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:29.474999 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:29.475440 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:29.475442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:29.475504 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:29.476087 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:29.476108 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:29.478118 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":112,"threads_started":1}
W20260929 17:58:29.478477 16818 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:29.472634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:44321 (local address 127.7.118.62:42997)
0929 17:58:29.472947 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:29.472952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:29.472968 (+ 16us) server_negotiation.cc:407] Connection header received
0929 17:58:29.473019 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:29.473021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:29.473064 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:29.473153 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:29.473611 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:29.474280 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:29.475103 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:29.475334 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:29.475419 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:29.478332 (+ 2913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.1:44321: BlockingRecv error: recv got EOF from 127.7.118.1:44321 (error 108)
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":90,"threads_started":1}
Sep 29 17:58:29 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:29 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:29.703125 16819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:29.696974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.2:58405)
0929 17:58:29.697271 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:29.697285 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:29.697372 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:29.697632 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:29.697634 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:29.697817 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:29.697909 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:29.697914 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:29.698791 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:29.698795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:29.699678 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:29.699696 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:29.700402 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:29.700406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:29.700484 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:29.700996 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:29.701009 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:29.703006 (+ 1997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":118,"threads_started":1}
W20260929 17:58:29.703275 16818 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:29.697252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:58405 (local address 127.7.118.62:42997)
0929 17:58:29.697402 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:29.697406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:29.697418 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:58:29.697463 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:29.697465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:29.697507 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:29.697593 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:29.698017 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:29.698645 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:29.699937 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:29.700277 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:29.700363 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:29.703151 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.2:58405: BlockingRecv error: recv got EOF from 127.7.118.2:58405 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 17:58:29 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:29 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:29.943174 16820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:29.936853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.3:46715)
0929 17:58:29.937175 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:29.937199 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:29.937347 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:29.937730 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:29.937734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:29.938098 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:29.938202 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:29.938209 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:29.939104 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:29.939108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:29.939781 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:29.939791 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:29.940323 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:29.940326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:29.940409 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:29.941022 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:29.941035 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:29.942979 (+ 1944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":118,"threads_started":1}
W20260929 17:58:29.943331 16818 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:29.936926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:46715 (local address 127.7.118.62:42997)
0929 17:58:29.937142 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:29.937148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:29.937282 (+ 134us) server_negotiation.cc:407] Connection header received
0929 17:58:29.937496 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:29.937498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:29.937555 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:29.937708 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:29.938414 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:29.938981 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:29.939915 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:29.940175 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:29.940276 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:29.943194 (+ 2918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.3:46715: BlockingRecv error: recv got EOF from 127.7.118.3:46715 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 17:58:30 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:30 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:30.485414 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:30.479318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.1:52471)
0929 17:58:30.479602 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:30.479621 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:30.479730 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:30.480028 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:30.480031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:30.480247 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:30.480341 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:30.480347 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:30.481215 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:30.481218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:30.481801 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:30.481809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:30.482404 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:30.482407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:30.482475 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:30.483069 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:30.483081 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:30.485225 (+ 2144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":94,"threads_started":1}
W20260929 17:58:30.485500 16822 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:30.479461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:52471 (local address 127.7.118.62:42997)
0929 17:58:30.479717 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:30.479721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:30.479738 (+ 17us) server_negotiation.cc:407] Connection header received
0929 17:58:30.479828 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:30.479831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:30.479875 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:30.479956 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:30.480507 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:30.481105 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:30.481930 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:30.482185 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:30.482309 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:30.485373 (+ 3064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.1:52471: BlockingRecv error: recv got EOF from 127.7.118.1:52471 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":104,"threads_started":1}
Sep 29 17:58:30 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:30 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:30.710072 16823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:30.704163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.2:55485)
0929 17:58:30.704583 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:30.704603 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:30.704734 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:30.705016 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:30.705019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:30.705257 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:30.705388 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:30.705398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:30.706180 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:30.706183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:30.706887 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:30.706897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:30.707377 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:30.707380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:30.707457 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:30.707953 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:30.707965 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:30.709948 (+ 1983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":86,"threads_started":1}
W20260929 17:58:30.710469 16822 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:30.704286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:55485 (local address 127.7.118.62:42997)
0929 17:58:30.704446 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:30.704452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:30.704761 (+ 309us) server_negotiation.cc:407] Connection header received
0929 17:58:30.704817 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:30.704821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:30.704880 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:30.704957 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:30.705527 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:30.706068 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:30.707018 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:30.707246 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:30.707330 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:30.710135 (+ 2805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.2:55485: BlockingRecv error: recv got EOF from 127.7.118.2:55485 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 17:58:30 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:30 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:30.950949 16824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:30.944356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.3:33347)
0929 17:58:30.944743 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:30.944775 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:30.944924 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:30.945252 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:30.945255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:30.945470 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:30.945559 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:30.945576 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:30.946503 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:30.946507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:30.947272 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:30.947282 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:30.947813 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:30.947816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:30.947903 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:30.948439 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:30.948452 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:30.950751 (+ 2299us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":89,"threads_started":1}
W20260929 17:58:30.951030 16822 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:30.944442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:33347 (local address 127.7.118.62:42997)
0929 17:58:30.944626 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:30.944632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:30.944858 (+ 226us) server_negotiation.cc:407] Connection header received
0929 17:58:30.945032 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:30.945036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:30.945122 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:30.945224 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:30.945693 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:30.946396 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:30.947395 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:30.947698 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:30.947809 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:30.950913 (+ 3104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.3:33347: BlockingRecv error: recv got EOF from 127.7.118.3:33347 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 29 17:58:31 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:31 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:31.492996 16825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:31.486450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.1:53487)
0929 17:58:31.486789 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:31.486803 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:31.486908 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:31.487208 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:31.487212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:31.487470 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:31.487613 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:31.487623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:31.488822 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:31.488826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:31.489406 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:31.489416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:31.489984 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:31.489987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:31.490060 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:31.490740 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:31.490753 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:31.492817 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":99,"threads_started":1}
W20260929 17:58:31.493116 16826 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:31.486486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:53487 (local address 127.7.118.62:42997)
0929 17:58:31.486780 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:31.486784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:31.486876 (+ 92us) server_negotiation.cc:407] Connection header received
0929 17:58:31.486988 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:31.486990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:31.487035 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:31.487116 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:31.487811 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:31.488693 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:31.489567 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:31.489855 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:31.489961 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:31.492978 (+ 3017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.1:53487: BlockingRecv error: recv got EOF from 127.7.118.1:53487 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":98,"threads_started":1}
Sep 29 17:58:31 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:31 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:31.717587 16827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:31.711153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.2:54515)
0929 17:58:31.711637 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:31.711652 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:31.711802 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:31.712091 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:31.712094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:31.712321 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:31.712413 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:31.712419 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:31.713194 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:31.713200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:31.713804 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:31.713814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:31.714348 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:31.714352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:31.714439 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:31.715083 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:31.715096 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:31.717407 (+ 2311us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":92,"threads_started":1}
W20260929 17:58:31.717657 16826 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:31.711248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:54515 (local address 127.7.118.62:42997)
0929 17:58:31.711561 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:31.711566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:31.711743 (+ 177us) server_negotiation.cc:407] Connection header received
0929 17:58:31.711916 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:31.711919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:31.711973 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:31.712068 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:31.712540 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:31.713074 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:31.713933 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:31.714187 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:31.714341 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:31.717547 (+ 3206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.2:54515: BlockingRecv error: recv got EOF from 127.7.118.2:54515 (error 108)
Metrics: {"server-negotiator.queue_time_us":197}
Sep 29 17:58:31 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:31 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:31.957669 16828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:31.951882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.3:57725)
0929 17:58:31.952201 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:31.952214 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:31.952321 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:31.952555 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:31.952557 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:31.952742 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:31.952838 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:31.952843 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:31.953665 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:31.953668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:31.954278 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:31.954289 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:31.954810 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:31.954812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:31.954882 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:31.955443 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:31.955500 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:31.957546 (+ 2046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":93,"threads_started":1}
W20260929 17:58:31.957944 16826 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:31.951969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:57725 (local address 127.7.118.62:42997)
0929 17:58:31.952111 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:31.952115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:31.952329 (+ 214us) server_negotiation.cc:407] Connection header received
0929 17:58:31.952387 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:31.952389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:31.952440 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:31.952564 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:31.952947 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:31.953478 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:31.954422 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:31.954683 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:31.954779 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:31.957820 (+ 3041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.3:57725: BlockingRecv error: recv got EOF from 127.7.118.3:57725 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 17:58:32 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:32 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:32.500031 16829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:32.493945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.1:39727)
0929 17:58:32.494361 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:32.494397 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:32.494524 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:32.494863 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:32.494871 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:32.495067 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:32.495180 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:32.495187 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:32.495943 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:32.495946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:32.496564 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:32.496571 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:32.497040 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:32.497043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:32.497117 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:32.497716 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:32.497729 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:32.499901 (+ 2172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":86,"threads_started":1}
W20260929 17:58:32.500243 16830 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:32.494086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:39727 (local address 127.7.118.62:42997)
0929 17:58:32.494402 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:32.494407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:32.494469 (+ 62us) server_negotiation.cc:407] Connection header received
0929 17:58:32.494639 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:32.494645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:32.494704 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:32.494809 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:32.495300 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:32.495843 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:32.496688 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:32.496920 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:32.497003 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:32.500088 (+ 3085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.1:39727: BlockingRecv error: recv got EOF from 127.7.118.1:39727 (error 108)
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":108,"threads_started":1}
Sep 29 17:58:32 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:32 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:32.725243 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:32.718714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.2:58849)
0929 17:58:32.719003 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:32.719023 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:32.719137 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:32.719490 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:32.719493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:32.719745 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:32.719850 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:32.719856 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:32.720896 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:32.720899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:32.721467 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:32.721484 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:32.722032 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:32.722036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:32.722126 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:32.722764 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:32.722777 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:32.725102 (+ 2325us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":120,"threads_started":1}
W20260929 17:58:32.725468 16830 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:32.718876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:58849 (local address 127.7.118.62:42997)
0929 17:58:32.719017 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:32.719021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:32.719105 (+ 84us) server_negotiation.cc:407] Connection header received
0929 17:58:32.719273 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:32.719276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:32.719326 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:32.719414 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:32.720009 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:32.720749 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:32.721617 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:32.721871 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:32.721972 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:32.725313 (+ 3341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.2:58849: BlockingRecv error: recv got EOF from 127.7.118.2:58849 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 17:58:32 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:32 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:32.965425 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:32.958946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.3:37041)
0929 17:58:32.959261 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:32.959275 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:32.959396 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:32.959771 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:32.959775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:32.959956 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:32.960064 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:32.960070 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:32.960920 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:32.960923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:32.961549 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:32.961559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:32.962164 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:32.962168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:32.962318 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:32.962994 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:32.963007 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:32.965297 (+ 2290us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":71,"threads_started":1}
W20260929 17:58:32.965633 16830 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:32.959088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:37041 (local address 127.7.118.62:42997)
0929 17:58:32.959276 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:32.959280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:32.959497 (+ 217us) server_negotiation.cc:407] Connection header received
0929 17:58:32.959554 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:32.959556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:32.959615 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:32.959695 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:32.960222 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:32.960786 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:32.961703 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:32.961951 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:32.962055 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:32.965455 (+ 3400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.3:37041: BlockingRecv error: recv got EOF from 127.7.118.3:37041 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 29 17:58:33 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:33 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:33.513785 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:33.500958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.1:43749)
0929 17:58:33.501349 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:33.501364 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:33.501484 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:33.501709 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:33.501711 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:33.501942 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:33.502038 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:33.502044 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:33.502848 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:33.502850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:33.503465 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:33.503475 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:33.504008 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:33.504011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:33.504081 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:33.511427 (+ 7346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:33.511446 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:33.513657 (+ 2211us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":108,"threads_started":1}
W20260929 17:58:33.513962 16834 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:33.501055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:43749 (local address 127.7.118.62:42997)
0929 17:58:33.501369 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:33.501373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:33.501421 (+ 48us) server_negotiation.cc:407] Connection header received
0929 17:58:33.501575 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:33.501577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:33.501620 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:33.501696 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:33.502141 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:33.502728 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:33.503612 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:33.503846 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:33.503939 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:33.513811 (+ 9872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.1:43749: BlockingRecv error: recv got EOF from 127.7.118.1:43749 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":61,"threads_started":1}
Sep 29 17:58:33 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:33 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:33.733781 16835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:33.726430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.2:52451)
0929 17:58:33.726844 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:33.726864 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:33.727006 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:33.727344 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:33.727347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:33.727540 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:33.727653 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:33.727664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:33.728481 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:33.728484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:33.729313 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:33.729326 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:33.729940 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:33.729947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:33.730124 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:33.730957 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:33.730971 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:33.733647 (+ 2676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":173,"threads_started":1}
W20260929 17:58:33.733971 16834 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:33.726496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:52451 (local address 127.7.118.62:42997)
0929 17:58:33.726759 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:33.726763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:33.727006 (+ 243us) server_negotiation.cc:407] Connection header received
0929 17:58:33.727091 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:33.727093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:33.727145 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:33.727249 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:33.727811 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:33.728348 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:33.729476 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:33.729794 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:33.729895 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:33.733822 (+ 3927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.2:52451: BlockingRecv error: recv got EOF from 127.7.118.2:52451 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 29 17:58:33 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:33 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:33.972874 16836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:33.966536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.3:37231)
0929 17:58:33.966871 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:33.966884 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:33.966991 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:33.967303 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:33.967306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:33.967487 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:33.967594 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:33.967600 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:33.968429 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:33.968432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:33.969019 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:33.969028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:33.969635 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:33.969639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:33.969721 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:33.970457 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:33.970471 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:33.972708 (+ 2237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":103,"threads_started":1}
W20260929 17:58:33.973083 16834 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:33.966591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:37231 (local address 127.7.118.62:42997)
0929 17:58:33.966725 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:33.966732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:33.966999 (+ 267us) server_negotiation.cc:407] Connection header received
0929 17:58:33.967096 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:33.967101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:33.967177 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:33.967276 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:33.967746 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:33.968305 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:33.969160 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:33.969476 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:33.969578 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:33.972910 (+ 3332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.3:37231: BlockingRecv error: recv got EOF from 127.7.118.3:37231 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 29 17:58:34 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:34 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:34.520569 16837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:34.514828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.1:53903)
0929 17:58:34.515135 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:34.515159 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:34.515256 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:34.515530 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:34.515532 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:34.515716 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:34.515808 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:34.515814 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:34.516635 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:34.516639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:34.517213 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:34.517222 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:34.517744 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:34.517747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:34.517816 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:34.518446 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:34.518458 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:34.520445 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":70,"threads_started":1}
W20260929 17:58:34.520849 16838 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:34.514982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:53903 (local address 127.7.118.62:42997)
0929 17:58:34.515293 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:34.515296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:34.515311 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:58:34.515353 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:34.515356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:34.515397 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:34.515495 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:34.515945 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:34.516505 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:34.517374 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:34.517620 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:34.517717 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:34.520696 (+ 2979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.1:53903: BlockingRecv error: recv got EOF from 127.7.118.1:53903 (error 108)
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":77,"threads_started":1}
Sep 29 17:58:34 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:34 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:34.741037 16839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:34.734869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.2:44649)
0929 17:58:34.735253 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:34.735272 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:34.735421 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:34.735718 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:34.735721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:34.735947 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:34.736050 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:34.736056 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:34.736830 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:34.736836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:34.737508 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:34.737517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:34.737994 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:34.737998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:34.738078 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:34.738702 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:34.738715 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:34.740865 (+ 2150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":101,"threads_started":1}
W20260929 17:58:34.741181 16838 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:34.735011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:44649 (local address 127.7.118.62:42997)
0929 17:58:34.735220 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:34.735225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:34.735364 (+ 139us) server_negotiation.cc:407] Connection header received
0929 17:58:34.735536 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:34.735540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:34.735600 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:34.735697 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:34.736175 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:34.736711 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:34.737639 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:34.737870 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:34.737986 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:34.741066 (+ 3080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.2:44649: BlockingRecv error: recv got EOF from 127.7.118.2:44649 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 29 17:58:34 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:34 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:34.980970 16840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:34.973985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.3:53539)
0929 17:58:34.974357 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:34.974372 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:34.974459 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:34.974732 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:34.974735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:34.974934 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:34.975069 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:34.975076 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:34.975976 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:34.975980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:34.976615 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:34.976626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:34.977152 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:34.977156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:34.977325 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:34.978341 (+ 1016us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:34.978367 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:34.980760 (+ 2393us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":125,"threads_started":1}
W20260929 17:58:34.981163 16838 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:34.974215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:53539 (local address 127.7.118.62:42997)
0929 17:58:34.974375 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:34.974379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:34.974489 (+ 110us) server_negotiation.cc:407] Connection header received
0929 17:58:34.974539 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:34.974542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:34.974586 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:34.974657 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:34.975232 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:34.975813 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:34.976767 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:34.976996 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:34.977118 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:34.980997 (+ 3879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.3:53539: BlockingRecv error: recv got EOF from 127.7.118.3:53539 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 17:58:35 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:35 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:35.528110 16841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:35.521553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.1:54755)
0929 17:58:35.521819 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:35.521832 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:35.521921 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:35.522231 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:35.522234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:35.522528 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:35.522656 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:35.522663 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:35.523542 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:35.523545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:35.524224 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:35.524237 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:35.524875 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:35.524878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:35.524953 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:35.525514 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:35.525530 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:35.527965 (+ 2435us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":96,"threads_started":1}
W20260929 17:58:35.528363 16842 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:35.521648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:54755 (local address 127.7.118.62:42997)
0929 17:58:35.521898 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:35.521902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:35.521916 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:58:35.522004 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:35.522006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:35.522077 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:35.522161 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:35.522799 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:35.523396 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:35.524403 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:35.524721 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:35.524829 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:35.528178 (+ 3349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.1:54755: BlockingRecv error: recv got EOF from 127.7.118.1:54755 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":97,"threads_started":1}
Sep 29 17:58:35 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:35 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:35.748472 16843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:35.742026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.2:59155)
0929 17:58:35.742412 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:35.742429 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:35.742600 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:35.743003 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:35.743006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:35.743262 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:35.743393 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:35.743401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:35.744223 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:35.744226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:35.744898 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:35.744909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:35.745515 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:35.745518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:35.745640 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:35.746289 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:35.746303 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:35.748316 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":82,"threads_started":1}
W20260929 17:58:35.748742 16842 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:35.742116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:59155 (local address 127.7.118.62:42997)
0929 17:58:35.742280 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:35.742286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:35.742546 (+ 260us) server_negotiation.cc:407] Connection header received
0929 17:58:35.742722 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:35.742726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:35.742789 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:35.742913 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:35.743517 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:35.744076 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:35.745080 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:35.745345 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:35.745448 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:35.748481 (+ 3033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.2:59155: BlockingRecv error: recv got EOF from 127.7.118.2:59155 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 17:58:35 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:35 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:35.988682 16844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:35.982110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.3:49833)
0929 17:58:35.982419 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:35.982436 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:35.982585 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:35.982941 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:35.982943 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:35.983117 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:35.983226 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:35.983231 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:35.984120 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:35.984123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:35.984706 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:35.984715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:35.985257 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:35.985261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:35.985345 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:35.985990 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:35.986008 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:35.988528 (+ 2520us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":149,"threads_started":1}
W20260929 17:58:35.988881 16842 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:35.982358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:49833 (local address 127.7.118.62:42997)
0929 17:58:35.982498 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:35.982501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:35.982517 (+ 16us) server_negotiation.cc:407] Connection header received
0929 17:58:35.982738 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:35.982741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:35.982792 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:35.982917 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:35.983429 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:35.983992 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:35.984842 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:35.985106 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:35.985204 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:35.988719 (+ 3515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.3:49833: BlockingRecv error: recv got EOF from 127.7.118.3:49833 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 17:58:36 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:36 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:36.535681 16845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:36.529066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.1:50843)
0929 17:58:36.529390 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:36.529450 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:36.529552 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:36.529872 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:36.529875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:36.530081 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:36.530193 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:36.530199 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:36.531421 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:36.531424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:36.532067 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:36.532082 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:36.532644 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:36.532647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:36.532717 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:36.533250 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:36.533262 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:36.535536 (+ 2274us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":123,"threads_started":1}
W20260929 17:58:36.535892 16846 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:36.529235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:50843 (local address 127.7.118.62:42997)
0929 17:58:36.529493 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:36.529496 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:36.529510 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:58:36.529633 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:36.529635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:36.529678 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:36.529765 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:36.530371 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:36.531279 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:36.532216 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:36.532458 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:36.532600 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:36.535724 (+ 3124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.1:50843: BlockingRecv error: recv got EOF from 127.7.118.1:50843 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":60,"threads_started":1}
Sep 29 17:58:36 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:36 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:36.756096 16847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:36.749278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.2:39155)
0929 17:58:36.749738 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:36.749763 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:36.749916 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:36.750523 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:36.750525 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:36.750752 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:36.750857 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:36.750864 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:36.751718 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:36.751725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:36.752391 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:36.752401 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:36.753064 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:36.753069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:36.753166 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:36.753743 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:36.753757 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:36.755950 (+ 2193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":86,"threads_started":1}
W20260929 17:58:36.756326 16846 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:36.749399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:39155 (local address 127.7.118.62:42997)
0929 17:58:36.749579 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:36.749590 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:36.749857 (+ 267us) server_negotiation.cc:407] Connection header received
0929 17:58:36.750124 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:36.750130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:36.750197 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:36.750315 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:36.751002 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:36.751592 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:36.752557 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:36.752925 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:36.753056 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:36.756168 (+ 3112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.2:39155: BlockingRecv error: recv got EOF from 127.7.118.2:39155 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 17:58:36 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:36 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:36.997336 16848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:36.989697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.3:50109)
0929 17:58:36.990061 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:36.990081 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:36.990221 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:36.990600 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:36.990606 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:36.990884 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:36.991048 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:36.991056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:36.992230 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:36.992234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:36.993171 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:36.993183 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:36.993855 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:36.993858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:36.993946 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:36.994614 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:36.994636 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:36.997205 (+ 2569us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":95,"threads_started":1}
W20260929 17:58:36.997612 16846 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:36.989793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:50109 (local address 127.7.118.62:42997)
0929 17:58:36.990054 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:36.990061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:36.990156 (+ 95us) server_negotiation.cc:407] Connection header received
0929 17:58:36.990370 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:36.990376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:36.990453 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:36.990552 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:36.991164 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:36.992094 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:36.993299 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:36.993709 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:36.993848 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:36.997480 (+ 3632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.3:50109: BlockingRecv error: recv got EOF from 127.7.118.3:50109 (error 108)
Metrics: {"server-negotiator.queue_time_us":155}
Sep 29 17:58:37 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:37 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:37.542698 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:37.536609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.1:47017)
0929 17:58:37.536951 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:37.536965 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:37.537117 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:37.537394 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:37.537396 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:37.537578 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:37.537672 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:37.537677 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:37.538665 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:37.538668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:37.539260 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:37.539270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:37.539762 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:37.539765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:37.539831 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:37.540414 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:37.540429 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:37.542566 (+ 2137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":74,"threads_started":1}
W20260929 17:58:37.542905 16850 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:37.536706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:47017 (local address 127.7.118.62:42997)
0929 17:58:37.536970 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:37.536973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:37.537118 (+ 145us) server_negotiation.cc:407] Connection header received
0929 17:58:37.537220 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:37.537222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:37.537271 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:37.537359 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:37.537781 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:37.538546 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:37.539398 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:37.539645 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:37.539738 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:37.542747 (+ 3009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.1:47017: BlockingRecv error: recv got EOF from 127.7.118.1:47017 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":101,"threads_started":1}
Sep 29 17:58:37 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:37 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:37.762800 16851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:37.757025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.2:46937)
0929 17:58:37.757276 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:37.757289 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:37.757374 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:37.757607 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:37.757608 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:37.757786 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:37.757878 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:37.757883 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:37.758683 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:37.758686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:37.759282 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:37.759292 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:37.759861 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:37.759864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:37.759928 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:37.760488 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:37.760501 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:37.762644 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":82,"threads_started":1}
W20260929 17:58:37.762969 16850 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:37.757167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:46937 (local address 127.7.118.62:42997)
0929 17:58:37.757332 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:37.757335 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:37.757349 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:58:37.757469 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:37.757471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:37.757520 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:37.757598 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:37.757983 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:37.758546 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:37.759459 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:37.759742 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:37.759856 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:37.762834 (+ 2978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.2:46937: BlockingRecv error: recv got EOF from 127.7.118.2:46937 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 17:58:38 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:38 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:38.005707 16852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:37.998477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.3:55369)
0929 17:58:37.998962 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:37.998983 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:37.999183 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:37.999543 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:37.999546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:37.999780 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:37.999888 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:37.999895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:38.000757 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:38.000763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:38.001602 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:38.001615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:38.002309 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:38.002314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:38.002421 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:38.003119 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:38.003141 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:38.005584 (+ 2443us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":170,"threads_started":1}
W20260929 17:58:38.005940 16850 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:37.998503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:55369 (local address 127.7.118.62:42997)
0929 17:58:37.998827 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:37.998832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:37.999105 (+ 273us) server_negotiation.cc:407] Connection header received
0929 17:58:37.999336 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:37.999340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:37.999409 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:37.999517 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:38.000034 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:38.000624 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:38.001750 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:38.002069 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:38.002203 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:38.005779 (+ 3576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.3:55369: BlockingRecv error: recv got EOF from 127.7.118.3:55369 (error 108)
Metrics: {"server-negotiator.queue_time_us":224}
Sep 29 17:58:38 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:38 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:38.554183 16853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:38.543754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.1:51595)
0929 17:58:38.544030 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:38.544052 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:38.544156 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:38.544415 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:38.544417 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:38.544619 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:38.544717 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:38.544723 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:38.545562 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:38.545565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:38.546223 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:38.546233 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:38.546940 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:38.546944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:38.547036 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:38.552047 (+ 5011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:38.552065 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:38.554054 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":100,"threads_started":1}
W20260929 17:58:38.554450 16854 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:38.543823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:51595 (local address 127.7.118.62:42997)
0929 17:58:38.544145 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:38.544149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:38.544166 (+ 17us) server_negotiation.cc:407] Connection header received
0929 17:58:38.544230 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:38.544232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:38.544276 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:38.544365 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:38.544836 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:38.545427 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:38.546408 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:38.546751 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:38.546863 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:38.554294 (+ 7431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.1:51595: BlockingRecv error: recv got EOF from 127.7.118.1:51595 (error 108)
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":125,"threads_started":1}
Sep 29 17:58:38 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:38 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:38.770603 16855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:38.763870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.2:48871)
0929 17:58:38.764433 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:38.764448 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:38.764578 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:38.764856 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:38.764858 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:38.765083 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:38.765192 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:38.765230 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:38.766154 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:38.766157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:38.766936 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:38.766947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:38.767521 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:38.767524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:38.767596 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:38.768226 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:38.768239 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:38.770470 (+ 2231us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":160,"threads_started":1}
W20260929 17:58:38.770831 16854 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:38.764077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:48871 (local address 127.7.118.62:42997)
0929 17:58:38.764258 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:38.764266 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:38.764558 (+ 292us) server_negotiation.cc:407] Connection header received
0929 17:58:38.764669 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:38.764671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:38.764723 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:38.764802 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:38.765431 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:38.766019 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:38.767090 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:38.767345 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:38.767485 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:38.770677 (+ 3192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.2:48871: BlockingRecv error: recv got EOF from 127.7.118.2:48871 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 17:58:39 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:39 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.3@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:39.013592 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:39.006823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.3:49577)
0929 17:58:39.007257 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:39.007277 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:39.007452 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:39.007799 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:39.007802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:39.008037 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:39.008142 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:39.008149 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:39.009018 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:39.009024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:39.009702 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:39.009713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:39.010314 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:39.010319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:39.010414 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:39.011100 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:39.011114 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:39.013388 (+ 2274us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":115,"threads_started":1}
W20260929 17:58:39.013653 16854 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:39.007001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:49577 (local address 127.7.118.62:42997)
0929 17:58:39.007227 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:39.007233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:39.007382 (+ 149us) server_negotiation.cc:407] Connection header received
0929 17:58:39.007596 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:39.007599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:39.007667 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:39.007769 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:39.008284 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:39.008885 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:39.009857 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:39.010126 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:39.010303 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:39.013534 (+ 3231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.3:49577: BlockingRecv error: recv got EOF from 127.7.118.3:49577 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Sep 29 17:58:39 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:39 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.1@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:39.562037 16857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:39.555372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.1:57723)
0929 17:58:39.555726 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:39.555751 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:39.555849 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:39.556157 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:39.556159 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:39.556344 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:39.556448 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:39.556454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:39.557753 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:39.557757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:39.558546 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:39.558557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:39.559131 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:39.559134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:39.559204 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:39.559831 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:39.559842 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:39.561885 (+ 2043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":78,"threads_started":1}
W20260929 17:58:39.562201 16858 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:39.555516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:57723 (local address 127.7.118.62:42997)
0929 17:58:39.555826 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:39.555830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:39.555847 (+ 17us) server_negotiation.cc:407] Connection header received
0929 17:58:39.555974 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:39.555977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:39.556024 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:39.556113 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:39.556630 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:39.557538 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:39.558720 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:39.558996 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:39.559091 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:39.562058 (+ 2967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.1:57723: BlockingRecv error: recv got EOF from 127.7.118.1:57723 (error 108)
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":88,"threads_started":1}
Sep 29 17:58:39 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:58:39 dist-test-slave-r7ph krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.118.2@KRBTEST.COM for oryx/127.7.118.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:58:39.777920 16859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:39.771626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42997 (local address 127.7.118.2:52625)
0929 17:58:39.771979 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:39.771999 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:39.772154 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:39.772555 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:39.772558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:39.772779 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:39.772882 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:39.772888 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:39.773876 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:39.773882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:39.774597 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:39.774608 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:39.775191 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:39.775194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:39.775318 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:39.775853 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:39.775869 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:39.777798 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.118.62:42997: Server oryx/127.7.118.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":103,"threads_started":1}
W20260929 17:58:39.778070 16858 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:39.771843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:52625 (local address 127.7.118.62:42997)
0929 17:58:39.772029 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:39.772036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:39.772106 (+ 70us) server_negotiation.cc:407] Connection header received
0929 17:58:39.772282 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:39.772286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:39.772392 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:39.772511 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:39.773019 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:39.773747 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:39.774763 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:39.775037 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:39.775150 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:39.777951 (+ 2801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.118.2:52625: BlockingRecv error: recv got EOF from 127.7.118.2:52625 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 17:58:39.882926 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 16383
I20260929 17:58:39.889062 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 16535
I20260929 17:58:39.897138 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 16667
I20260929 17:58:39.903525 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 16313
2026-09-29T17:58:39Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18361 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T17:58:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:58:39Z Disabled control of system clock
I20260929 17:58:39.936467 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:36281
--webserver_interface=127.7.118.62
--webserver_port=0
--builtin_ntp_servers=127.7.118.20:38049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:36281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 17:58:40.055639 16863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:40.055970 16863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:40.056008 16863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:40.060519 16863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:58:40.060609 16863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:40.060624 16863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:40.060642 16863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:58:40.060658 16863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:58:40.066143 16863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:38049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:36281
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:36281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:40.067462 16863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:40.069025 16863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:40.079033 16871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:40.079033 16869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:40.079033 16868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:40.080058 16863 server_base.cc:1034] running on GCE node
I20260929 17:58:40.080763 16863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:40.082038 16863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:40.083258 16863 hybrid_clock.cc:648] HybridClock initialized: now 1790704720083233 us; error 46 us; skew 500 ppm
I20260929 17:58:40.085536 16863 webserver.cc:460] Webserver started at http://127.7.118.62:42323/ using document root <none> and password file <none>
I20260929 17:58:40.086356 16863 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:40.086479 16863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:40.086750 16863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:40.089136 16863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "18008ba3b1b948e68d48434ccc668cde"
format_stamp: "Formatted at 2026-09-29 17:58:40 on dist-test-slave-r7ph"
server_key: "f77699cc236e7117aceadd34709467e7"
server_key_iv: "fe710fc7630498359984f9699616a1c6"
server_key_version: "encryptionkey@0"
I20260929 17:58:40.089835 16863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "18008ba3b1b948e68d48434ccc668cde"
format_stamp: "Formatted at 2026-09-29 17:58:40 on dist-test-slave-r7ph"
server_key: "f77699cc236e7117aceadd34709467e7"
server_key_iv: "fe710fc7630498359984f9699616a1c6"
server_key_version: "encryptionkey@0"
I20260929 17:58:40.094626 16863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260929 17:58:40.097625 16877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:40.099327 16863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:58:40.099565 16863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "18008ba3b1b948e68d48434ccc668cde"
format_stamp: "Formatted at 2026-09-29 17:58:40 on dist-test-slave-r7ph"
server_key: "f77699cc236e7117aceadd34709467e7"
server_key_iv: "fe710fc7630498359984f9699616a1c6"
server_key_version: "encryptionkey@0"
I20260929 17:58:40.099727 16863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:40.114007 16863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:40.114888 16863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:40.115118 16863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:40.122913 16863 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:36281
I20260929 17:58:40.122984 16929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:36281 every 8 connection(s)
I20260929 17:58:40.124073 16863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 17:58:40.127852 16930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:40.134040 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 16863
I20260929 17:58:40.134167 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 17:58:40.134579 7640 external_mini_cluster.cc:1428] Setting key dd5cb3e609445b3d86c0f71e5abe4dcd
I20260929 17:58:40.134945 16930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18008ba3b1b948e68d48434ccc668cde: Bootstrap starting.
I20260929 17:58:40.137753 16930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18008ba3b1b948e68d48434ccc668cde: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:40.138736 16930 log.cc:824] T 00000000000000000000000000000000 P 18008ba3b1b948e68d48434ccc668cde: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:40.141073 16930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18008ba3b1b948e68d48434ccc668cde: No bootstrap required, opened a new log
I20260929 17:58:40.145529 16933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:40.136137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47634 (local address 127.7.118.62:36281)
0929 17:58:40.136938 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:40.136956 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:40.136997 (+ 41us) server_negotiation.cc:407] Connection header received
0929 17:58:40.137726 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:40.137747 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:40.138383 (+ 636us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:40.138774 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:40.139283 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:40.140363 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:40.141482 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:40.142053 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:40.142241 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:40.143251 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:40.143256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:40.143271 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:58:40.143748 (+ 477us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:40.143812 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:58:40.143833 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:40.143953 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:40.144124 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 17:58:40.144419 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":178,"threads_started":1}
I20260929 17:58:40.145362 16930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18008ba3b1b948e68d48434ccc668cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18008ba3b1b948e68d48434ccc668cde" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 36281 } }
I20260929 17:58:40.145747 16930 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18008ba3b1b948e68d48434ccc668cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:40.145891 16930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18008ba3b1b948e68d48434ccc668cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18008ba3b1b948e68d48434ccc668cde, State: Initialized, Role: FOLLOWER
I20260929 17:58:40.146490 16930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18008ba3b1b948e68d48434ccc668cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18008ba3b1b948e68d48434ccc668cde" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 36281 } }
I20260929 17:58:40.146672 16930 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18008ba3b1b948e68d48434ccc668cde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:58:40.146763 16930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18008ba3b1b948e68d48434ccc668cde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:58:40.146878 16930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18008ba3b1b948e68d48434ccc668cde [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:40.148206 16930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18008ba3b1b948e68d48434ccc668cde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18008ba3b1b948e68d48434ccc668cde" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 36281 } }
I20260929 17:58:40.148649 16930 leader_election.cc:302] T 00000000000000000000000000000000 P 18008ba3b1b948e68d48434ccc668cde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18008ba3b1b948e68d48434ccc668cde; no voters:
I20260929 17:58:40.148979 16930 leader_election.cc:288] T 00000000000000000000000000000000 P 18008ba3b1b948e68d48434ccc668cde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:58:40.149173 16935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18008ba3b1b948e68d48434ccc668cde [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:58:40.149442 16935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18008ba3b1b948e68d48434ccc668cde [term 1 LEADER]: Becoming Leader. State: Replica: 18008ba3b1b948e68d48434ccc668cde, State: Running, Role: LEADER
I20260929 17:58:40.149789 16935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18008ba3b1b948e68d48434ccc668cde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18008ba3b1b948e68d48434ccc668cde" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 36281 } }
I20260929 17:58:40.150417 16930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18008ba3b1b948e68d48434ccc668cde [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:58:40.151933 16937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18008ba3b1b948e68d48434ccc668cde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18008ba3b1b948e68d48434ccc668cde. Latest consensus state: current_term: 1 leader_uuid: "18008ba3b1b948e68d48434ccc668cde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18008ba3b1b948e68d48434ccc668cde" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 36281 } } }
I20260929 17:58:40.152123 16937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18008ba3b1b948e68d48434ccc668cde [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:40.152170 16936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18008ba3b1b948e68d48434ccc668cde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18008ba3b1b948e68d48434ccc668cde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18008ba3b1b948e68d48434ccc668cde" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 36281 } } }
I20260929 17:58:40.152300 16936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18008ba3b1b948e68d48434ccc668cde [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:40.152873 16946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:58:40.156414 16946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:58:40.163089 16946 catalog_manager.cc:1347] Generated new cluster ID: 8f7fe4c1d48c4db9a9c043d2a7afa74f
I20260929 17:58:40.163201 16946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:58:40.173246 16946 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:58:40.174376 16946 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:58:40.180341 16946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18008ba3b1b948e68d48434ccc668cde: Generated new TSK 0
I20260929 17:58:40.181177 16946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:58:40.184616 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:0
--local_ip_for_outbound_sockets=127.7.118.1
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:36281
--builtin_ntp_servers=127.7.118.20:38049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 17:58:40.303722 16954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:40.304034 16954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:40.304100 16954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:40.308686 16954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:40.308769 16954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:40.308830 16954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 17:58:40.314213 16954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:38049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:36281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:40.315469 16954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:40.316998 16954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:40.328164 16962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:40.328161 16960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:40.328164 16959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:40.328271 16954 server_base.cc:1034] running on GCE node
I20260929 17:58:40.328974 16954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:40.329695 16954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:40.330892 16954 hybrid_clock.cc:648] HybridClock initialized: now 1790704720330872 us; error 36 us; skew 500 ppm
I20260929 17:58:40.333267 16954 webserver.cc:460] Webserver started at http://127.7.118.1:41577/ using document root <none> and password file <none>
I20260929 17:58:40.333966 16954 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:40.334069 16954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:40.334376 16954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:40.336689 16954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/data/instance:
uuid: "03b597df594b4c7483f687628af7ad19"
format_stamp: "Formatted at 2026-09-29 17:58:40 on dist-test-slave-r7ph"
server_key: "207fe19f56d40ab22a90c5c074ddae40"
server_key_iv: "98d370c1efbd2beb7e08686f6ce4face"
server_key_version: "encryptionkey@0"
I20260929 17:58:40.337355 16954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance:
uuid: "03b597df594b4c7483f687628af7ad19"
format_stamp: "Formatted at 2026-09-29 17:58:40 on dist-test-slave-r7ph"
server_key: "207fe19f56d40ab22a90c5c074ddae40"
server_key_iv: "98d370c1efbd2beb7e08686f6ce4face"
server_key_version: "encryptionkey@0"
I20260929 17:58:40.341645 16954 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 17:58:40.344552 16968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:40.346091 16954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:58:40.346312 16954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "03b597df594b4c7483f687628af7ad19"
format_stamp: "Formatted at 2026-09-29 17:58:40 on dist-test-slave-r7ph"
server_key: "207fe19f56d40ab22a90c5c074ddae40"
server_key_iv: "98d370c1efbd2beb7e08686f6ce4face"
server_key_version: "encryptionkey@0"
I20260929 17:58:40.346480 16954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:40.393307 16954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:40.394161 16954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:40.394447 16954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:40.395138 16954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:40.396246 16954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:40.396329 16954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:40.396418 16954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:40.396473 16954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:40.407394 16954 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:44943
I20260929 17:58:40.407495 17081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:44943 every 8 connection(s)
I20260929 17:58:40.408529 16954 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 17:58:40.412391 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 16954
I20260929 17:58:40.412540 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance
I20260929 17:58:40.412846 7640 external_mini_cluster.cc:1428] Setting key 0a55cbb57cfe209800baefea5ef7846a
I20260929 17:58:40.414970 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:0
--local_ip_for_outbound_sockets=127.7.118.2
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:36281
--builtin_ntp_servers=127.7.118.20:38049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 17:58:40.419866 16933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:40.410700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:37465 (local address 127.7.118.62:36281)
0929 17:58:40.410852 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:40.410856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:40.412239 (+ 1383us) server_negotiation.cc:407] Connection header received
0929 17:58:40.414092 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:40.414095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:40.414154 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:40.414236 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:40.415432 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:40.416031 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:40.417395 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:40.417659 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:40.417793 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:40.419208 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:40.419212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:40.419214 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:58:40.419230 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:40.419262 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:58:40.419271 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:40.419443 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:40.419629 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 17:58:40.419702 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 17:58:40.420902 17084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:40.411174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:36281 (local address 127.7.118.1:37465)
0929 17:58:40.412065 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:40.412129 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:40.413838 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:40.414479 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:40.414488 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:40.414591 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:40.415272 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:40.415289 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:40.416200 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:40.416209 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:40.417236 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:40.417247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:40.417860 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:40.417865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:40.418063 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:40.418879 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:58:40.418881 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:40.418933 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:58:40.419013 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:40.419444 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:40.419458 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:40.419468 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 17:58:40.419631 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 17:58:40.420000 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":177,"threads_started":1}
I20260929 17:58:40.422390 17082 heartbeater.cc:347] Connected to a master server at 127.7.118.62:36281
I20260929 17:58:40.422685 17082 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:40.423300 17082 heartbeater.cc:511] Master 127.7.118.62:36281 requested a full tablet report, sending...
I20260929 17:58:40.425191 16893 ts_manager.cc:194] Registered new tserver with Master: 03b597df594b4c7483f687628af7ad19 (127.7.118.1:44943)
I20260929 17:58:40.426873 16893 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.118.1:37465
W20260929 17:58:40.539433 17085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:40.539752 17085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:40.539790 17085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:40.544194 17085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:40.544286 17085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:40.544345 17085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 17:58:40.549472 17085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:38049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:36281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:40.550762 17085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:40.552443 17085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:58:40.563088 17085 server_base.cc:1034] running on GCE node
W20260929 17:58:40.563028 17090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:40.563041 17091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:40.563022 17093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:40.563894 17085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:40.564720 17085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:40.565940 17085 hybrid_clock.cc:648] HybridClock initialized: now 1790704720565916 us; error 39 us; skew 500 ppm
I20260929 17:58:40.568176 17085 webserver.cc:460] Webserver started at http://127.7.118.2:36845/ using document root <none> and password file <none>
I20260929 17:58:40.568833 17085 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:40.568943 17085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:40.569197 17085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:40.571547 17085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/data/instance:
uuid: "e5649fcd1f2a4e4aac1ab4c83e30f55e"
format_stamp: "Formatted at 2026-09-29 17:58:40 on dist-test-slave-r7ph"
server_key: "28c729548c300b01d913c312becf7147"
server_key_iv: "3d5116569ab88e0064e45b85b6ef41af"
server_key_version: "encryptionkey@0"
I20260929 17:58:40.572216 17085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance:
uuid: "e5649fcd1f2a4e4aac1ab4c83e30f55e"
format_stamp: "Formatted at 2026-09-29 17:58:40 on dist-test-slave-r7ph"
server_key: "28c729548c300b01d913c312becf7147"
server_key_iv: "3d5116569ab88e0064e45b85b6ef41af"
server_key_version: "encryptionkey@0"
I20260929 17:58:40.576692 17085 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:58:40.579857 17099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:40.581346 17085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 17:58:40.581624 17085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "e5649fcd1f2a4e4aac1ab4c83e30f55e"
format_stamp: "Formatted at 2026-09-29 17:58:40 on dist-test-slave-r7ph"
server_key: "28c729548c300b01d913c312becf7147"
server_key_iv: "3d5116569ab88e0064e45b85b6ef41af"
server_key_version: "encryptionkey@0"
I20260929 17:58:40.581779 17085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:40.625665 17085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:40.626684 17085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:40.626909 17085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:40.627565 17085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:40.628625 17085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:40.628703 17085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:40.628779 17085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:40.628831 17085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:40.639767 17085 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:41293
I20260929 17:58:40.639914 17212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:41293 every 8 connection(s)
I20260929 17:58:40.641103 17085 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 17:58:40.642856 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 17085
I20260929 17:58:40.642987 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance
I20260929 17:58:40.643332 7640 external_mini_cluster.cc:1428] Setting key 02ed037ea61a212bf339e93894e55b6d
I20260929 17:58:40.645372 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:0
--local_ip_for_outbound_sockets=127.7.118.3
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:36281
--builtin_ntp_servers=127.7.118.20:38049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 17:58:40.653496 16933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:40.643411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:58813 (local address 127.7.118.62:36281)
0929 17:58:40.643556 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:40.643561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:40.644866 (+ 1305us) server_negotiation.cc:407] Connection header received
0929 17:58:40.646496 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:40.646503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:40.646574 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:40.646677 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:40.648028 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:40.648605 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:40.650830 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:40.651194 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:40.651286 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:40.652934 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:40.652938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:40.652941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:58:40.652956 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:40.652981 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:58:40.652991 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:40.653087 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:40.653301 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 17:58:40.653367 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 17:58:40.654999 17215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:40.643905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:36281 (local address 127.7.118.2:58813)
0929 17:58:40.644705 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:40.644765 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:40.646191 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:40.646913 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:40.646925 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:40.647056 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:40.647844 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:40.647869 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:40.648758 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:40.648765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:40.650597 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:40.650614 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:40.651381 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:40.651388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:40.651688 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:40.652523 (+ 835us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:58:40.652528 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:40.652604 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:58:40.652717 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:40.653108 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:40.653122 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:40.653130 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 17:58:40.653304 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 17:58:40.653693 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":158,"threads_started":1}
I20260929 17:58:40.656520 17213 heartbeater.cc:347] Connected to a master server at 127.7.118.62:36281
I20260929 17:58:40.656836 17213 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:40.657459 17213 heartbeater.cc:511] Master 127.7.118.62:36281 requested a full tablet report, sending...
I20260929 17:58:40.658692 16893 ts_manager.cc:194] Registered new tserver with Master: e5649fcd1f2a4e4aac1ab4c83e30f55e (127.7.118.2:41293)
I20260929 17:58:40.659413 16893 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.118.2:58813
W20260929 17:58:40.767676 17216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:40.767997 17216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:40.768036 17216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:40.772327 17216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:40.772405 17216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:40.772466 17216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 17:58:40.777552 17216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:38049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:36281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:40.778779 17216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:40.780202 17216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:40.790629 17222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:40.790622 17221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:40.790879 17216 server_base.cc:1034] running on GCE node
W20260929 17:58:40.790622 17224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:40.791548 17216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:40.792313 17216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:40.793511 17216 hybrid_clock.cc:648] HybridClock initialized: now 1790704720793484 us; error 43 us; skew 500 ppm
I20260929 17:58:40.795874 17216 webserver.cc:460] Webserver started at http://127.7.118.3:43813/ using document root <none> and password file <none>
I20260929 17:58:40.796509 17216 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:40.796581 17216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:40.796767 17216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:40.799182 17216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/data/instance:
uuid: "b903df1ca35d43fcbb040816bf49efb3"
format_stamp: "Formatted at 2026-09-29 17:58:40 on dist-test-slave-r7ph"
server_key: "78b47e5f292442e8c62bee12b74e4872"
server_key_iv: "1cfce75208a3b95f996ce7b1fe7c556d"
server_key_version: "encryptionkey@0"
I20260929 17:58:40.799805 17216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance:
uuid: "b903df1ca35d43fcbb040816bf49efb3"
format_stamp: "Formatted at 2026-09-29 17:58:40 on dist-test-slave-r7ph"
server_key: "78b47e5f292442e8c62bee12b74e4872"
server_key_iv: "1cfce75208a3b95f996ce7b1fe7c556d"
server_key_version: "encryptionkey@0"
I20260929 17:58:40.804253 17216 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:58:40.807327 17230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:40.808725 17216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:58:40.808864 17216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "b903df1ca35d43fcbb040816bf49efb3"
format_stamp: "Formatted at 2026-09-29 17:58:40 on dist-test-slave-r7ph"
server_key: "78b47e5f292442e8c62bee12b74e4872"
server_key_iv: "1cfce75208a3b95f996ce7b1fe7c556d"
server_key_version: "encryptionkey@0"
I20260929 17:58:40.809001 17216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:40.823048 17216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:40.823832 17216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:40.824056 17216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:40.824731 17216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:40.825778 17216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:40.825826 17216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:40.825894 17216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:40.825928 17216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:40.835695 17343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:45271 every 8 connection(s)
I20260929 17:58:40.835673 17216 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:45271
I20260929 17:58:40.836799 17216 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 17:58:40.842890 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 17216
I20260929 17:58:40.843034 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance
I20260929 17:58:40.843389 7640 external_mini_cluster.cc:1428] Setting key 529e5475030e68c2ec01c4389d646258
I20260929 17:58:40.847710 16933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:40.838901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:59317 (local address 127.7.118.62:36281)
0929 17:58:40.839081 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:40.839085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:40.840309 (+ 1224us) server_negotiation.cc:407] Connection header received
0929 17:58:40.841744 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:40.841747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:40.841822 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:40.841900 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:40.843267 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:40.843957 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:40.845428 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:40.845781 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:40.845943 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:40.847170 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:40.847174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:40.847176 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:58:40.847190 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:40.847208 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:58:40.847217 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:40.847288 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:40.847528 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 17:58:40.847589 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 17:58:40.848685 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:40.839371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:36281 (local address 127.7.118.3:59317)
0929 17:58:40.840098 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:40.840157 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:40.841492 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:40.842141 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:40.842149 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:40.842242 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:40.843087 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:40.843113 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:40.844101 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:40.844106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:40.845242 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:40.845253 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:40.845979 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:40.845983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:40.846140 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:40.846875 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:58:40.846876 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:40.846925 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:58:40.847000 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:40.847341 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:40.847349 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:40.847354 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 17:58:40.847479 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 17:58:40.847748 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":118,"threads_started":1}
I20260929 17:58:40.850049 17344 heartbeater.cc:347] Connected to a master server at 127.7.118.62:36281
I20260929 17:58:40.850593 17344 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:40.851362 17344 heartbeater.cc:511] Master 127.7.118.62:36281 requested a full tablet report, sending...
I20260929 17:58:40.852836 16893 ts_manager.cc:194] Registered new tserver with Master: b903df1ca35d43fcbb040816bf49efb3 (127.7.118.3:45271)
I20260929 17:58:40.853487 16893 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.118.3:59317
I20260929 17:58:40.858377 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 17:58:40.861369 16933 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:40.860628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47638 (local address 127.7.118.62:36281)
0929 17:58:40.860770 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:40.860774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:40.860789 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:58:40.860882 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:40.860884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:40.860929 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:40.861010 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:40.861243 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47638: BlockingRecv error: recv got EOF from 127.0.0.1:47638 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 17:58:40.861617 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 16954
I20260929 17:58:40.867898 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 17085
I20260929 17:58:40.873587 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 17216
I20260929 17:58:40.879073 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 16863
2026-09-29T17:58:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (974 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T17:58:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:58:40Z Disabled control of system clock
I20260929 17:58:40.909228 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:40049
--webserver_interface=127.7.118.62
--webserver_port=0
--builtin_ntp_servers=127.7.118.20:35777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:40049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 17:58:41.030102 17355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:41.030627 17355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:41.030699 17355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:41.035157 17355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:58:41.035243 17355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:41.035259 17355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:41.035277 17355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:58:41.035295 17355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:58:41.040477 17355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:35777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:40049
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:40049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:41.041708 17355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:41.043371 17355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:41.053532 17361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:41.053615 17355 server_base.cc:1034] running on GCE node
W20260929 17:58:41.053409 17360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:41.053478 17363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:41.054543 17355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:41.055954 17355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:41.057235 17355 hybrid_clock.cc:648] HybridClock initialized: now 1790704721057130 us; error 89 us; skew 500 ppm
I20260929 17:58:41.059612 17355 webserver.cc:460] Webserver started at http://127.7.118.62:38855/ using document root <none> and password file <none>
I20260929 17:58:41.060292 17355 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:41.060400 17355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:41.060771 17355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:41.063450 17355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "995f57ee1e12457c97e444adae926ec6"
format_stamp: "Formatted at 2026-09-29 17:58:41 on dist-test-slave-r7ph"
server_key: "c38262b5ffe271658d474ef13b2e7990"
server_key_iv: "bc2d4e4e0c692fc602711c17296e30d7"
server_key_version: "encryptionkey@0"
I20260929 17:58:41.064146 17355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "995f57ee1e12457c97e444adae926ec6"
format_stamp: "Formatted at 2026-09-29 17:58:41 on dist-test-slave-r7ph"
server_key: "c38262b5ffe271658d474ef13b2e7990"
server_key_iv: "bc2d4e4e0c692fc602711c17296e30d7"
server_key_version: "encryptionkey@0"
I20260929 17:58:41.068589 17355 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:58:41.071535 17369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:41.073534 17355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:58:41.073750 17355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "995f57ee1e12457c97e444adae926ec6"
format_stamp: "Formatted at 2026-09-29 17:58:41 on dist-test-slave-r7ph"
server_key: "c38262b5ffe271658d474ef13b2e7990"
server_key_iv: "bc2d4e4e0c692fc602711c17296e30d7"
server_key_version: "encryptionkey@0"
I20260929 17:58:41.074115 17355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:41.079422 17355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:41.080159 17355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:41.080374 17355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:41.088393 17355 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:40049
I20260929 17:58:41.088423 17421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:40049 every 8 connection(s)
I20260929 17:58:41.089504 17355 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 17:58:41.093292 17422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:41.096073 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 17355
I20260929 17:58:41.096222 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 17:58:41.096550 7640 external_mini_cluster.cc:1428] Setting key e9a8489fd5c85b4fa76d64db110453ba
I20260929 17:58:41.100250 17422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 995f57ee1e12457c97e444adae926ec6: Bootstrap starting.
I20260929 17:58:41.102998 17422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 995f57ee1e12457c97e444adae926ec6: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:41.103015 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:41.098228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50012 (local address 127.7.118.62:40049)
0929 17:58:41.099076 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:41.099094 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:41.099132 (+ 38us) server_negotiation.cc:407] Connection header received
0929 17:58:41.099919 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:41.099940 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:41.100355 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:41.100737 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:41.101564 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:41.101568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:41.101583 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:58:41.101593 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:41.101648 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:58:41.101664 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:41.101841 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:41.101928 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 17:58:41.102185 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":152,"threads_started":1}
I20260929 17:58:41.103837 17422 log.cc:824] T 00000000000000000000000000000000 P 995f57ee1e12457c97e444adae926ec6: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:41.106158 17422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 995f57ee1e12457c97e444adae926ec6: No bootstrap required, opened a new log
I20260929 17:58:41.110167 17422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 995f57ee1e12457c97e444adae926ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995f57ee1e12457c97e444adae926ec6" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 40049 } }
I20260929 17:58:41.110532 17422 raft_consensus.cc:374] T 00000000000000000000000000000000 P 995f57ee1e12457c97e444adae926ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:41.110625 17422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 995f57ee1e12457c97e444adae926ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 995f57ee1e12457c97e444adae926ec6, State: Initialized, Role: FOLLOWER
I20260929 17:58:41.111155 17422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 995f57ee1e12457c97e444adae926ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995f57ee1e12457c97e444adae926ec6" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 40049 } }
I20260929 17:58:41.111327 17422 raft_consensus.cc:388] T 00000000000000000000000000000000 P 995f57ee1e12457c97e444adae926ec6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:58:41.111418 17422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 995f57ee1e12457c97e444adae926ec6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:58:41.111522 17422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 995f57ee1e12457c97e444adae926ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:41.112557 17422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 995f57ee1e12457c97e444adae926ec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995f57ee1e12457c97e444adae926ec6" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 40049 } }
I20260929 17:58:41.112948 17422 leader_election.cc:302] T 00000000000000000000000000000000 P 995f57ee1e12457c97e444adae926ec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 995f57ee1e12457c97e444adae926ec6; no voters:
I20260929 17:58:41.113299 17422 leader_election.cc:288] T 00000000000000000000000000000000 P 995f57ee1e12457c97e444adae926ec6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:58:41.113545 17427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 995f57ee1e12457c97e444adae926ec6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:58:41.113791 17427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 995f57ee1e12457c97e444adae926ec6 [term 1 LEADER]: Becoming Leader. State: Replica: 995f57ee1e12457c97e444adae926ec6, State: Running, Role: LEADER
I20260929 17:58:41.114188 17427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 995f57ee1e12457c97e444adae926ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995f57ee1e12457c97e444adae926ec6" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 40049 } }
I20260929 17:58:41.114835 17422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 995f57ee1e12457c97e444adae926ec6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:58:41.116053 17429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 995f57ee1e12457c97e444adae926ec6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 995f57ee1e12457c97e444adae926ec6. Latest consensus state: current_term: 1 leader_uuid: "995f57ee1e12457c97e444adae926ec6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995f57ee1e12457c97e444adae926ec6" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 40049 } } }
I20260929 17:58:41.116206 17429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 995f57ee1e12457c97e444adae926ec6 [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:41.116498 17428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 995f57ee1e12457c97e444adae926ec6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "995f57ee1e12457c97e444adae926ec6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995f57ee1e12457c97e444adae926ec6" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 40049 } } }
I20260929 17:58:41.116595 17428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 995f57ee1e12457c97e444adae926ec6 [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:41.116609 17436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:58:41.119551 17436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:58:41.125633 17436 catalog_manager.cc:1347] Generated new cluster ID: 95d69b57a4fd4e1c857060f1d0de6c1f
I20260929 17:58:41.125735 17436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:58:41.134081 17436 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:58:41.134564 17436 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:58:41.140935 17436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 995f57ee1e12457c97e444adae926ec6: Generated new TSK 0
I20260929 17:58:41.141924 17436 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:58:41.152621 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:0
--local_ip_for_outbound_sockets=127.7.118.1
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:40049
--builtin_ntp_servers=127.7.118.20:35777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 17:58:41.273191 17446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:41.273514 17446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:41.273558 17446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:41.278112 17446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:41.278198 17446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:41.278309 17446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 17:58:41.283689 17446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:35777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:40049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:41.284951 17446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:41.286484 17446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:41.297013 17452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:41.297046 17451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:41.297016 17454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:41.297473 17446 server_base.cc:1034] running on GCE node
I20260929 17:58:41.298029 17446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:41.298929 17446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:41.300169 17446 hybrid_clock.cc:648] HybridClock initialized: now 1790704721300118 us; error 67 us; skew 500 ppm
I20260929 17:58:41.302462 17446 webserver.cc:460] Webserver started at http://127.7.118.1:34615/ using document root <none> and password file <none>
I20260929 17:58:41.303119 17446 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:41.303227 17446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:41.303493 17446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:41.305719 17446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/data/instance:
uuid: "379736bd8dd24dc08c2d4a3d7d946dd1"
format_stamp: "Formatted at 2026-09-29 17:58:41 on dist-test-slave-r7ph"
server_key: "1deb842d10aaaadb6cbee94c6e9f0220"
server_key_iv: "cb30a61e639a29459526ebfc88d7c2c6"
server_key_version: "encryptionkey@0"
I20260929 17:58:41.306447 17446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance:
uuid: "379736bd8dd24dc08c2d4a3d7d946dd1"
format_stamp: "Formatted at 2026-09-29 17:58:41 on dist-test-slave-r7ph"
server_key: "1deb842d10aaaadb6cbee94c6e9f0220"
server_key_iv: "cb30a61e639a29459526ebfc88d7c2c6"
server_key_version: "encryptionkey@0"
I20260929 17:58:41.310694 17446 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:58:41.313611 17460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:41.315223 17446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:58:41.315478 17446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "379736bd8dd24dc08c2d4a3d7d946dd1"
format_stamp: "Formatted at 2026-09-29 17:58:41 on dist-test-slave-r7ph"
server_key: "1deb842d10aaaadb6cbee94c6e9f0220"
server_key_iv: "cb30a61e639a29459526ebfc88d7c2c6"
server_key_version: "encryptionkey@0"
I20260929 17:58:41.315620 17446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:41.320859 17446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:41.321589 17446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:41.321803 17446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:41.322531 17446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:41.323570 17446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:41.323642 17446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:41.323709 17446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:41.323760 17446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:41.333515 17446 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:37897
I20260929 17:58:41.333532 17573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:37897 every 8 connection(s)
I20260929 17:58:41.334671 17446 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 17:58:41.340315 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 17446
I20260929 17:58:41.340489 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance
I20260929 17:58:41.340823 7640 external_mini_cluster.cc:1428] Setting key 37c1ae073a8080f14694c36644b5280a
I20260929 17:58:41.341678 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:41.336727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:33623 (local address 127.7.118.62:40049)
0929 17:58:41.336875 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:41.336879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:41.338087 (+ 1208us) server_negotiation.cc:407] Connection header received
0929 17:58:41.339628 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:41.339631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:41.339681 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:41.339762 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:41.341082 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:41.341086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:41.341089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:58:41.341103 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:41.341133 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:58:41.341143 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:41.341255 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:41.341484 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 17:58:41.341528 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 17:58:41.343014 17576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:41.337135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:40049 (local address 127.7.118.1:33623)
0929 17:58:41.337919 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:41.337965 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:41.339374 (+ 1409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:41.339962 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:41.339970 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:41.340098 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:41.340751 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:58:41.340753 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:41.340797 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:58:41.340882 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:41.341276 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:41.341285 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:41.341291 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:58:41.341520 (+ 229us) client_negotiation.cc:240] Negotiation successful
0929 17:58:41.341825 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":155,"threads_started":1}
I20260929 17:58:41.343088 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:0
--local_ip_for_outbound_sockets=127.7.118.2
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:40049
--builtin_ntp_servers=127.7.118.20:35777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 17:58:41.344544 17574 heartbeater.cc:347] Connected to a master server at 127.7.118.62:40049
I20260929 17:58:41.344887 17574 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:41.345527 17574 heartbeater.cc:511] Master 127.7.118.62:40049 requested a full tablet report, sending...
I20260929 17:58:41.347530 17386 ts_manager.cc:194] Registered new tserver with Master: 379736bd8dd24dc08c2d4a3d7d946dd1 (127.7.118.1:37897)
W20260929 17:58:41.460426 17577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:41.460708 17577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:41.460752 17577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:41.464977 17577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:41.465050 17577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:41.465111 17577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 17:58:41.470886 17577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:35777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:40049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:41.472098 17577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:41.473592 17577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:58:41.484467 17577 server_base.cc:1034] running on GCE node
W20260929 17:58:41.484423 17585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:41.484396 17582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:41.484426 17583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:41.485114 17577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:41.485842 17577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:41.487048 17577 hybrid_clock.cc:648] HybridClock initialized: now 1790704721487000 us; error 62 us; skew 500 ppm
I20260929 17:58:41.489252 17577 webserver.cc:460] Webserver started at http://127.7.118.2:37081/ using document root <none> and password file <none>
I20260929 17:58:41.489876 17577 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:41.489945 17577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:41.490175 17577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:41.492424 17577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/data/instance:
uuid: "1a3e2985028a492d84fb63db4d211299"
format_stamp: "Formatted at 2026-09-29 17:58:41 on dist-test-slave-r7ph"
server_key: "958ad7270e3f24d4b1e393f57b5c724a"
server_key_iv: "a5a36d2bb9f0cd080b68f2d1ebf3a3c5"
server_key_version: "encryptionkey@0"
I20260929 17:58:41.493078 17577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance:
uuid: "1a3e2985028a492d84fb63db4d211299"
format_stamp: "Formatted at 2026-09-29 17:58:41 on dist-test-slave-r7ph"
server_key: "958ad7270e3f24d4b1e393f57b5c724a"
server_key_iv: "a5a36d2bb9f0cd080b68f2d1ebf3a3c5"
server_key_version: "encryptionkey@0"
I20260929 17:58:41.497843 17577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 17:58:41.502133 17591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:41.504273 17577 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260929 17:58:41.504446 17577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "1a3e2985028a492d84fb63db4d211299"
format_stamp: "Formatted at 2026-09-29 17:58:41 on dist-test-slave-r7ph"
server_key: "958ad7270e3f24d4b1e393f57b5c724a"
server_key_iv: "a5a36d2bb9f0cd080b68f2d1ebf3a3c5"
server_key_version: "encryptionkey@0"
I20260929 17:58:41.504698 17577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:41.510396 17577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:41.511209 17577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:41.511435 17577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:41.512076 17577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:41.513245 17577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:41.513298 17577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:41.513345 17577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:41.513362 17577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:41.523913 17577 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:42749
I20260929 17:58:41.523962 17704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:42749 every 8 connection(s)
I20260929 17:58:41.524959 17577 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 17:58:41.530938 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 17577
I20260929 17:58:41.531082 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance
I20260929 17:58:41.531411 7640 external_mini_cluster.cc:1428] Setting key bfa0fd0d24150efe9bc9b9df51765860
I20260929 17:58:41.532220 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:41.527218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:45993 (local address 127.7.118.62:40049)
0929 17:58:41.527544 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:41.527548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:41.528670 (+ 1122us) server_negotiation.cc:407] Connection header received
0929 17:58:41.530011 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:41.530015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:41.530057 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:41.530144 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:41.531680 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:41.531683 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:41.531685 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:58:41.531700 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:41.531746 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:58:41.531755 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:41.531868 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:41.532019 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 17:58:41.532072 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228}
I20260929 17:58:41.533330 17707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:41.527672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:40049 (local address 127.7.118.2:45993)
0929 17:58:41.528499 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:41.528544 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:41.529762 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:41.530469 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:41.530477 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:41.530611 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:41.531314 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:58:41.531316 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:41.531362 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:58:41.531440 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:41.531846 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:41.531855 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:41.531861 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:58:41.532069 (+ 208us) client_negotiation.cc:240] Negotiation successful
0929 17:58:41.532375 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":132,"threads_started":1}
I20260929 17:58:41.533654 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:0
--local_ip_for_outbound_sockets=127.7.118.3
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:40049
--builtin_ntp_servers=127.7.118.20:35777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 17:58:41.534767 17705 heartbeater.cc:347] Connected to a master server at 127.7.118.62:40049
I20260929 17:58:41.535069 17705 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:41.535557 17705 heartbeater.cc:511] Master 127.7.118.62:40049 requested a full tablet report, sending...
I20260929 17:58:41.536761 17386 ts_manager.cc:194] Registered new tserver with Master: 1a3e2985028a492d84fb63db4d211299 (127.7.118.2:42749)
W20260929 17:58:41.660866 17708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:41.661295 17708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:41.661379 17708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:41.665822 17708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:41.665941 17708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:41.666051 17708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 17:58:41.671702 17708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:35777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:40049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:41.673130 17708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:41.674714 17708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:58:41.686004 17708 server_base.cc:1034] running on GCE node
W20260929 17:58:41.685961 17716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:41.685961 17713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:41.685962 17714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:41.686700 17708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:41.687410 17708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:41.688603 17708 hybrid_clock.cc:648] HybridClock initialized: now 1790704721688577 us; error 45 us; skew 500 ppm
I20260929 17:58:41.690734 17708 webserver.cc:460] Webserver started at http://127.7.118.3:42535/ using document root <none> and password file <none>
I20260929 17:58:41.691363 17708 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:41.691488 17708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:41.691718 17708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:41.693919 17708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/data/instance:
uuid: "2488c88b97fc434bb3027ed626a41908"
format_stamp: "Formatted at 2026-09-29 17:58:41 on dist-test-slave-r7ph"
server_key: "8b296528f04bbaf3cc74616ef5e865b6"
server_key_iv: "e826e38c427859b7dabca81454f4956c"
server_key_version: "encryptionkey@0"
I20260929 17:58:41.694582 17708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance:
uuid: "2488c88b97fc434bb3027ed626a41908"
format_stamp: "Formatted at 2026-09-29 17:58:41 on dist-test-slave-r7ph"
server_key: "8b296528f04bbaf3cc74616ef5e865b6"
server_key_iv: "e826e38c427859b7dabca81454f4956c"
server_key_version: "encryptionkey@0"
I20260929 17:58:41.698725 17708 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 17:58:41.701753 17722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:41.703366 17708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:58:41.703588 17708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "2488c88b97fc434bb3027ed626a41908"
format_stamp: "Formatted at 2026-09-29 17:58:41 on dist-test-slave-r7ph"
server_key: "8b296528f04bbaf3cc74616ef5e865b6"
server_key_iv: "e826e38c427859b7dabca81454f4956c"
server_key_version: "encryptionkey@0"
I20260929 17:58:41.703737 17708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:41.709117 17708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:41.709939 17708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:41.710158 17708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:41.710974 17708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:41.711987 17708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:41.712071 17708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:41.712148 17708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:41.712180 17708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:41.722380 17708 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:39789
I20260929 17:58:41.722402 17835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:39789 every 8 connection(s)
I20260929 17:58:41.723415 17708 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 17:58:41.730312 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:41.725528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:38475 (local address 127.7.118.62:40049)
0929 17:58:41.725694 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:41.725699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:41.727065 (+ 1366us) server_negotiation.cc:407] Connection header received
0929 17:58:41.728414 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:41.728418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:41.728462 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:41.728545 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:41.729829 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:41.729832 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:41.729834 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:58:41.729852 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:41.729875 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:58:41.729883 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:41.729968 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:41.730100 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 17:58:41.730140 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 17:58:41.731283 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:41.726061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:40049 (local address 127.7.118.3:38475)
0929 17:58:41.726872 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:41.726919 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:41.728155 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:41.728739 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:41.728746 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:41.728854 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:41.729540 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:58:41.729542 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:41.729588 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:58:41.729667 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:41.729971 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:41.729977 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:41.729982 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 17:58:41.730126 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 17:58:41.730378 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":157,"threads_started":1}
I20260929 17:58:41.732475 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 17708
I20260929 17:58:41.732622 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance
I20260929 17:58:41.732574 17836 heartbeater.cc:347] Connected to a master server at 127.7.118.62:40049
I20260929 17:58:41.732916 17836 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:41.733011 7640 external_mini_cluster.cc:1428] Setting key a1034f02da6190d9e65e4b44dfc24f9c
I20260929 17:58:41.733448 17836 heartbeater.cc:511] Master 127.7.118.62:40049 requested a full tablet report, sending...
I20260929 17:58:41.734771 17385 ts_manager.cc:194] Registered new tserver with Master: 2488c88b97fc434bb3027ed626a41908 (127.7.118.3:39789)
I20260929 17:58:41.748080 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 17:58:41.751261 17425 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:58:41.750185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50020 (local address 127.7.118.62:40049)
0929 17:58:41.750372 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:41.750376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:41.750577 (+ 201us) server_negotiation.cc:407] Connection header received
0929 17:58:41.750742 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:41.750746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:41.750790 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:41.750883 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:41.751114 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50020: BlockingRecv error: recv got EOF from 127.0.0.1:50020 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 17:58:41.751768 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 17446
I20260929 17:58:41.757607 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 17577
I20260929 17:58:41.763259 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 17708
I20260929 17:58:41.769184 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 17355
2026-09-29T17:58:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (890 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:58:41 dist-test-slave-r7ph krb5kdc[17846](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:58:41 dist-test-slave-r7ph krb5kdc[17846](info): set up 2 sockets
Sep 29 17:58:41 dist-test-slave-r7ph krb5kdc[17846](info): commencing operation
krb5kdc: starting...
W20260929 17:58:42.839345 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 17:58:42 dist-test-slave-r7ph krb5kdc[17846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:58:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:58:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 17:58:42.976963 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:37689
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:37437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:37689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:43.100685 17859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:43.101066 17859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:43.101145 17859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:43.105468 17859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:58:43.105592 17859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:43.105629 17859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:43.105669 17859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:58:43.105710 17859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:58:43.111500 17859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:37437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:37689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:37689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:43.112788 17859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:43.113942 17859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:43.123739 17864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:43.123765 17867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:43.123893 17859 server_base.cc:1034] running on GCE node
W20260929 17:58:43.124157 17865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:43.124826 17859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:43.126139 17859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:43.127401 17859 hybrid_clock.cc:648] HybridClock initialized: now 1790704723127372 us; error 52 us; skew 500 ppm
Sep 29 17:58:43 dist-test-slave-r7ph krb5kdc[17846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704723, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:43.130586 17859 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 17:58:43.131686 17859 webserver.cc:460] Webserver started at http://127.7.118.62:42845/ using document root <none> and password file <none>
I20260929 17:58:43.132318 17859 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:43.132421 17859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:43.132663 17859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:43.135022 17859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "c410f626dd834ae08423d1549c677176"
format_stamp: "Formatted at 2026-09-29 17:58:43 on dist-test-slave-r7ph"
server_key: "c96b9a5302d08ca55e3759b22bcfba6f"
server_key_iv: "e30eac6804b5b88947d9fcbc251e0357"
server_key_version: "encryptionkey@0"
I20260929 17:58:43.135718 17859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "c410f626dd834ae08423d1549c677176"
format_stamp: "Formatted at 2026-09-29 17:58:43 on dist-test-slave-r7ph"
server_key: "c96b9a5302d08ca55e3759b22bcfba6f"
server_key_iv: "e30eac6804b5b88947d9fcbc251e0357"
server_key_version: "encryptionkey@0"
I20260929 17:58:43.140194 17859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:58:43.143363 17874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:43.145012 17859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:58:43.145207 17859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "c410f626dd834ae08423d1549c677176"
format_stamp: "Formatted at 2026-09-29 17:58:43 on dist-test-slave-r7ph"
server_key: "c96b9a5302d08ca55e3759b22bcfba6f"
server_key_iv: "e30eac6804b5b88947d9fcbc251e0357"
server_key_version: "encryptionkey@0"
I20260929 17:58:43.145368 17859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:43.166239 17859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:43.170506 17859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:43.170768 17859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:43.179423 17859 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:37689
I20260929 17:58:43.179729 17926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:37689 every 8 connection(s)
I20260929 17:58:43.180729 17859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 17:58:43.184083 17927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:43.185461 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 17859
I20260929 17:58:43.185591 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 17:58:43.185904 7640 external_mini_cluster.cc:1428] Setting key e341b07928faa68f741d739801e59045
I20260929 17:58:43.190963 17927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c410f626dd834ae08423d1549c677176: Bootstrap starting.
I20260929 17:58:43.193362 17927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c410f626dd834ae08423d1549c677176: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:43.194350 17927 log.cc:824] T 00000000000000000000000000000000 P c410f626dd834ae08423d1549c677176: Log is configured to *not* fsync() on all Append() calls
Sep 29 17:58:43 dist-test-slave-r7ph krb5kdc[17846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:58:43.196905 17927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c410f626dd834ae08423d1549c677176: No bootstrap required, opened a new log
I20260929 17:58:43.200714 17927 raft_consensus.cc:348] T 00000000000000000000000000000000 P c410f626dd834ae08423d1549c677176 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c410f626dd834ae08423d1549c677176" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 37689 } }
I20260929 17:58:43.201067 17927 raft_consensus.cc:374] T 00000000000000000000000000000000 P c410f626dd834ae08423d1549c677176 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:43.201164 17927 raft_consensus.cc:729] T 00000000000000000000000000000000 P c410f626dd834ae08423d1549c677176 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c410f626dd834ae08423d1549c677176, State: Initialized, Role: FOLLOWER
I20260929 17:58:43.201649 17927 consensus_queue.cc:260] T 00000000000000000000000000000000 P c410f626dd834ae08423d1549c677176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c410f626dd834ae08423d1549c677176" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 37689 } }
I20260929 17:58:43.201785 17927 raft_consensus.cc:388] T 00000000000000000000000000000000 P c410f626dd834ae08423d1549c677176 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:58:43.201872 17927 raft_consensus.cc:482] T 00000000000000000000000000000000 P c410f626dd834ae08423d1549c677176 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:58:43.201977 17927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c410f626dd834ae08423d1549c677176 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:43.202307 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:43.187621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60372 (local address 127.7.118.62:37689)
0929 17:58:43.188188 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:43.188205 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:43.188248 (+ 43us) server_negotiation.cc:407] Connection header received
0929 17:58:43.188987 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:43.189007 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:43.189398 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:43.189811 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:43.190782 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:43.191867 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:43.192865 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:43.193388 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:43.193575 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:43.196661 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:43.196667 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:43.196685 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:43.196717 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:43.198739 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:43.199283 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:43.199290 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:43.199296 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:43.199431 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:43.199752 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:43.199756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:43.199758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:43.200218 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:43.200453 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:43.200719 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 17:58:43.201105 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":175,"threads_started":1}
I20260929 17:58:43.203191 17927 raft_consensus.cc:504] T 00000000000000000000000000000000 P c410f626dd834ae08423d1549c677176 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c410f626dd834ae08423d1549c677176" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 37689 } }
I20260929 17:58:43.203580 17927 leader_election.cc:302] T 00000000000000000000000000000000 P c410f626dd834ae08423d1549c677176 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c410f626dd834ae08423d1549c677176; no voters:
I20260929 17:58:43.203950 17927 leader_election.cc:288] T 00000000000000000000000000000000 P c410f626dd834ae08423d1549c677176 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:58:43.204038 17932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c410f626dd834ae08423d1549c677176 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:58:43.204239 17932 raft_consensus.cc:686] T 00000000000000000000000000000000 P c410f626dd834ae08423d1549c677176 [term 1 LEADER]: Becoming Leader. State: Replica: c410f626dd834ae08423d1549c677176, State: Running, Role: LEADER
I20260929 17:58:43.204574 17932 consensus_queue.cc:237] T 00000000000000000000000000000000 P c410f626dd834ae08423d1549c677176 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c410f626dd834ae08423d1549c677176" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 37689 } }
I20260929 17:58:43.205420 17927 sys_catalog.cc:564] T 00000000000000000000000000000000 P c410f626dd834ae08423d1549c677176 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:58:43.205883 17932 sys_catalog.cc:455] T 00000000000000000000000000000000 P c410f626dd834ae08423d1549c677176 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c410f626dd834ae08423d1549c677176. Latest consensus state: current_term: 1 leader_uuid: "c410f626dd834ae08423d1549c677176" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c410f626dd834ae08423d1549c677176" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 37689 } } }
I20260929 17:58:43.206010 17932 sys_catalog.cc:458] T 00000000000000000000000000000000 P c410f626dd834ae08423d1549c677176 [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:43.206435 17933 sys_catalog.cc:455] T 00000000000000000000000000000000 P c410f626dd834ae08423d1549c677176 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c410f626dd834ae08423d1549c677176" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c410f626dd834ae08423d1549c677176" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 37689 } } }
I20260929 17:58:43.206532 17936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:58:43.206547 17933 sys_catalog.cc:458] T 00000000000000000000000000000000 P c410f626dd834ae08423d1549c677176 [sys.catalog]: This master's current role is: LEADER
I20260929 17:58:43.210052 17936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:58:43.216856 17936 catalog_manager.cc:1347] Generated new cluster ID: 196acc11dfa841b8a9e8dd4c6fd378fb
I20260929 17:58:43.216954 17936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:58:43.251336 17936 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:58:43.252734 17936 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:58:43.265017 17936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c410f626dd834ae08423d1549c677176: Generated new TSK 0
I20260929 17:58:43.266009 17936 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.118.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
I20260929 17:58:43.306849 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:0
--local_ip_for_outbound_sockets=127.7.118.1
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:37689
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:37437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:43.428373 17953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:43.428694 17953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:43.428938 17953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:43.434502 17953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:43.434587 17953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:43.434655 17953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 17:58:43.441015 17953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:37437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:37689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:43.442564 17953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:43.443722 17953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:58:43.455129 17953 server_base.cc:1034] running on GCE node
W20260929 17:58:43.455066 17958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:43.455066 17961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:43.455101 17959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:43.456071 17953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:43.456838 17953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:43.458024 17953 hybrid_clock.cc:648] HybridClock initialized: now 1790704723458005 us; error 35 us; skew 500 ppm
Sep 29 17:58:43 dist-test-slave-r7ph krb5kdc[17846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704723, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:43.461366 17953 init.cc:377] Logged in from keytab as kudu/127.7.118.1@KRBTEST.COM (short username kudu)
I20260929 17:58:43.462477 17953 webserver.cc:460] Webserver started at http://127.7.118.1:39745/ using document root <none> and password file <none>
I20260929 17:58:43.463106 17953 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:43.463204 17953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:43.463456 17953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:43.465672 17953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/data/instance:
uuid: "897ead4f78b64054902d0ef9148c6eaa"
format_stamp: "Formatted at 2026-09-29 17:58:43 on dist-test-slave-r7ph"
server_key: "726404479823f8848e456f110f45e4d3"
server_key_iv: "6a8bd4602fa0d13126ec4a8c62498da9"
server_key_version: "encryptionkey@0"
I20260929 17:58:43.466415 17953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance:
uuid: "897ead4f78b64054902d0ef9148c6eaa"
format_stamp: "Formatted at 2026-09-29 17:58:43 on dist-test-slave-r7ph"
server_key: "726404479823f8848e456f110f45e4d3"
server_key_iv: "6a8bd4602fa0d13126ec4a8c62498da9"
server_key_version: "encryptionkey@0"
I20260929 17:58:43.470786 17953 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 17:58:43.473981 17968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:43.475610 17953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:58:43.475787 17953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "897ead4f78b64054902d0ef9148c6eaa"
format_stamp: "Formatted at 2026-09-29 17:58:43 on dist-test-slave-r7ph"
server_key: "726404479823f8848e456f110f45e4d3"
server_key_iv: "6a8bd4602fa0d13126ec4a8c62498da9"
server_key_version: "encryptionkey@0"
I20260929 17:58:43.475936 17953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:43.492622 17953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:43.498118 17953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:43.498399 17953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:43.499298 17953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:43.500563 17953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:43.500631 17953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:43.500695 17953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:43.500717 17953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:43.510982 17953 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:36551
I20260929 17:58:43.511081 18081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:36551 every 8 connection(s)
I20260929 17:58:43.512097 17953 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 17:58:43.515321 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 17953
I20260929 17:58:43.515465 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance
I20260929 17:58:43.515796 7640 external_mini_cluster.cc:1428] Setting key 584e2e6db209d2aea46f453b256fcef9
Sep 29 17:58:43 dist-test-slave-r7ph krb5kdc[17846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704723, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.118.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.2@KRBTEST.COM" created.
I20260929 17:58:43.528447 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:43.514335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:39091 (local address 127.7.118.62:37689)
0929 17:58:43.514509 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:43.514513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:43.515661 (+ 1148us) server_negotiation.cc:407] Connection header received
0929 17:58:43.517182 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:43.517185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:43.517239 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:43.517320 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:43.518780 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:43.519340 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:43.520752 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:43.521038 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:43.521152 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:43.524270 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:43.524274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:43.524276 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:43.524304 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:43.526051 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:43.526772 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:43.526775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:43.526777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:43.526856 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:43.527233 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:43.527236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:43.527238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:43.527443 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:43.527555 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:43.528130 (+ 575us) server_negotiation.cc:299] Negotiation successful
0929 17:58:43.528278 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 17:58:43.529696 18084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:43.514778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:37689 (local address 127.7.118.1:39091)
0929 17:58:43.515395 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:43.515532 (+ 137us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:43.516944 (+ 1412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:43.517545 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:43.517553 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:43.517958 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:43.518608 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:43.518624 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:43.519490 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:43.519493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:43.520595 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:43.520606 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:43.521214 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:43.521218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:43.521410 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:43.522112 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:43.522129 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:43.524037 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:43.526195 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:43.526201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:43.526222 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:43.526645 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:43.526955 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:43.526959 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:43.526961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:43.527129 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:43.527565 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:43.527574 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:43.527868 (+ 294us) client_negotiation.cc:769] Sending connection context
0929 17:58:43.528074 (+ 206us) client_negotiation.cc:240] Negotiation successful
0929 17:58:43.528538 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":184,"threads_started":1}
I20260929 17:58:43.531462 18082 heartbeater.cc:347] Connected to a master server at 127.7.118.62:37689
I20260929 17:58:43.531827 18082 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:43.532415 18082 heartbeater.cc:511] Master 127.7.118.62:37689 requested a full tablet report, sending...
I20260929 17:58:43.534168 17891 ts_manager.cc:194] Registered new tserver with Master: 897ead4f78b64054902d0ef9148c6eaa (127.7.118.1:36551)
I20260929 17:58:43.535601 17891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.1@KRBTEST.COM'} at 127.7.118.1:39091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
I20260929 17:58:43.551301 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:0
--local_ip_for_outbound_sockets=127.7.118.2
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:37689
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:37437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:43.672400 18087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:43.672746 18087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:43.672828 18087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:43.677265 18087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:43.677392 18087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:43.677493 18087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 17:58:43.683408 18087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:37437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:37689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:43.684581 18087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:43.685688 18087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:43.696172 18093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:43.696188 18095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:43.696213 18092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:43.696728 18087 server_base.cc:1034] running on GCE node
I20260929 17:58:43.697237 18087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:43.698064 18087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:43.699291 18087 hybrid_clock.cc:648] HybridClock initialized: now 1790704723699252 us; error 63 us; skew 500 ppm
Sep 29 17:58:43 dist-test-slave-r7ph krb5kdc[17846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704723, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:43.708822 18087 init.cc:377] Logged in from keytab as kudu/127.7.118.2@KRBTEST.COM (short username kudu)
I20260929 17:58:43.709957 18087 webserver.cc:460] Webserver started at http://127.7.118.2:43459/ using document root <none> and password file <none>
I20260929 17:58:43.710656 18087 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:43.710752 18087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:43.711006 18087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:43.713222 18087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/data/instance:
uuid: "f74c96497d5a4587ac0d6e8e4ce6aba9"
format_stamp: "Formatted at 2026-09-29 17:58:43 on dist-test-slave-r7ph"
server_key: "30013fc13cb065f0ffdcdcf0fa4586c8"
server_key_iv: "6e947aa789bafc325c3045123ea56871"
server_key_version: "encryptionkey@0"
I20260929 17:58:43.713784 18087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance:
uuid: "f74c96497d5a4587ac0d6e8e4ce6aba9"
format_stamp: "Formatted at 2026-09-29 17:58:43 on dist-test-slave-r7ph"
server_key: "30013fc13cb065f0ffdcdcf0fa4586c8"
server_key_iv: "6e947aa789bafc325c3045123ea56871"
server_key_version: "encryptionkey@0"
I20260929 17:58:43.718001 18087 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260929 17:58:43.721045 18102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:43.722985 18087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:58:43.723163 18087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "f74c96497d5a4587ac0d6e8e4ce6aba9"
format_stamp: "Formatted at 2026-09-29 17:58:43 on dist-test-slave-r7ph"
server_key: "30013fc13cb065f0ffdcdcf0fa4586c8"
server_key_iv: "6e947aa789bafc325c3045123ea56871"
server_key_version: "encryptionkey@0"
I20260929 17:58:43.723311 18087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:43.736910 18087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:43.740615 18087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:43.740859 18087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:43.741503 18087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:43.742455 18087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:43.742527 18087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:43.742599 18087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:43.742640 18087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:43.752491 18087 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:39545
I20260929 17:58:43.752516 18215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:39545 every 8 connection(s)
I20260929 17:58:43.753525 18087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 17:58:43.759330 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 18087
I20260929 17:58:43.759465 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance
I20260929 17:58:43.759774 7640 external_mini_cluster.cc:1428] Setting key 1a2b15eb169a4fdad5f6f6dad06face2
Sep 29 17:58:43 dist-test-slave-r7ph krb5kdc[17846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704723, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.118.3@KRBTEST.COM; defaulting to no policy
I20260929 17:58:43.771265 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:43.755613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:43441 (local address 127.7.118.62:37689)
0929 17:58:43.755777 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:43.755781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:43.756772 (+ 991us) server_negotiation.cc:407] Connection header received
0929 17:58:43.758280 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:43.758284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:43.758345 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:43.758428 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:43.760027 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:43.760757 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:43.762544 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:43.762777 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:43.762872 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:43.766139 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:43.766144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:43.766147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:43.766181 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:43.768390 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:43.769055 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:43.769060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:43.769064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:43.769174 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:43.769748 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:43.769753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:43.769757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:43.770036 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:43.770162 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:43.770992 (+ 830us) server_negotiation.cc:299] Negotiation successful
0929 17:58:43.771134 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 17:58:43.772271 18218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:43.756009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:37689 (local address 127.7.118.2:43441)
0929 17:58:43.756589 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:43.756636 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:43.758020 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:43.758616 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:43.758624 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:43.759058 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:43.759829 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:43.759852 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:43.760900 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:43.760904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:43.762335 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:43.762351 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:43.762960 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:43.762967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:43.763246 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:43.764146 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:43.764164 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:43.765917 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:43.768579 (+ 2662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:43.768586 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:43.768602 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:43.768926 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:43.769318 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:43.769321 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:43.769322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:43.769613 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:43.770173 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:43.770179 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:43.770612 (+ 433us) client_negotiation.cc:769] Sending connection context
0929 17:58:43.770903 (+ 291us) client_negotiation.cc:240] Negotiation successful
0929 17:58:43.771252 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":123,"threads_started":1}
I20260929 17:58:43.773967 18216 heartbeater.cc:347] Connected to a master server at 127.7.118.62:37689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.3@KRBTEST.COM" created.
I20260929 17:58:43.774360 18216 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:43.774995 18216 heartbeater.cc:511] Master 127.7.118.62:37689 requested a full tablet report, sending...
I20260929 17:58:43.776440 17891 ts_manager.cc:194] Registered new tserver with Master: f74c96497d5a4587ac0d6e8e4ce6aba9 (127.7.118.2:39545)
I20260929 17:58:43.777094 17891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.2@KRBTEST.COM'} at 127.7.118.2:43441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
I20260929 17:58:43.795058 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:0
--local_ip_for_outbound_sockets=127.7.118.3
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:37689
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:37437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:58:43.915226 18221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:58:43.915623 18221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:58:43.915716 18221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:58:43.920521 18221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:58:43.920655 18221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:58:43.920755 18221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 17:58:43.926522 18221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:37437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:37689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:58:43.927850 18221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:58:43.929057 18221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:58:43.940063 18229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:43.940060 18227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:58:43.940054 18226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:58:43.940804 18221 server_base.cc:1034] running on GCE node
I20260929 17:58:43.941324 18221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:58:43.942077 18221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:58:43.943288 18221 hybrid_clock.cc:648] HybridClock initialized: now 1790704723943240 us; error 67 us; skew 500 ppm
Sep 29 17:58:43 dist-test-slave-r7ph krb5kdc[17846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704723, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:58:43.946471 18221 init.cc:377] Logged in from keytab as kudu/127.7.118.3@KRBTEST.COM (short username kudu)
I20260929 17:58:43.947446 18221 webserver.cc:460] Webserver started at http://127.7.118.3:45049/ using document root <none> and password file <none>
I20260929 17:58:43.948011 18221 fs_manager.cc:359] Metadata directory not provided
I20260929 17:58:43.948077 18221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:58:43.948254 18221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:58:43.950655 18221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/data/instance:
uuid: "281995707f1f453a85dfbb6e5901d6d9"
format_stamp: "Formatted at 2026-09-29 17:58:43 on dist-test-slave-r7ph"
server_key: "3c67409da02a3f28783aaa9b87df680c"
server_key_iv: "03a007d782641a916f3bad6d9daef7e0"
server_key_version: "encryptionkey@0"
I20260929 17:58:43.951218 18221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance:
uuid: "281995707f1f453a85dfbb6e5901d6d9"
format_stamp: "Formatted at 2026-09-29 17:58:43 on dist-test-slave-r7ph"
server_key: "3c67409da02a3f28783aaa9b87df680c"
server_key_iv: "03a007d782641a916f3bad6d9daef7e0"
server_key_version: "encryptionkey@0"
I20260929 17:58:43.955404 18221 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:58:43.958900 18236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:43.960388 18221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:58:43.960558 18221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "281995707f1f453a85dfbb6e5901d6d9"
format_stamp: "Formatted at 2026-09-29 17:58:43 on dist-test-slave-r7ph"
server_key: "3c67409da02a3f28783aaa9b87df680c"
server_key_iv: "03a007d782641a916f3bad6d9daef7e0"
server_key_version: "encryptionkey@0"
I20260929 17:58:43.960704 18221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:58:43.982232 18221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:58:43.986107 18221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:58:43.986361 18221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:58:43.987082 18221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:58:43.988016 18221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:58:43.988067 18221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:43.988117 18221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:58:43.988171 18221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:58:43.998172 18221 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:36569
I20260929 17:58:43.998198 18349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:36569 every 8 connection(s)
I20260929 17:58:43.999302 18221 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 17:58:44.002705 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 18221
I20260929 17:58:44.002883 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance
I20260929 17:58:44.003235 7640 external_mini_cluster.cc:1428] Setting key 164d6ab78a001502521080b1adf54226
Sep 29 17:58:44 dist-test-slave-r7ph krb5kdc[17846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704723, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:58:44.017897 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:44.001427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:38187 (local address 127.7.118.62:37689)
0929 17:58:44.001597 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:44.001603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:44.002807 (+ 1204us) server_negotiation.cc:407] Connection header received
0929 17:58:44.004987 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:44.004992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:44.005053 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:44.005140 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:44.006886 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.007512 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.008892 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.009139 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.009254 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:44.012814 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:44.012820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:44.012826 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:44.012867 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:44.015055 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:44.015814 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:44.015818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:44.015819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:44.015912 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:44.016357 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:44.016361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:44.016363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:44.016627 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:44.016739 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:44.017532 (+ 793us) server_negotiation.cc:299] Negotiation successful
0929 17:58:44.017704 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 17:58:44.018853 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:44.001916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:37689 (local address 127.7.118.3:38187)
0929 17:58:44.002592 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:44.002666 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:44.004700 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:44.005380 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:44.005389 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:44.006006 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:58:44.006712 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:44.006730 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.007642 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.007645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:44.008720 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:44.008733 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.009366 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.009369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:44.009552 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:44.010752 (+ 1200us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:58:44.010774 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:58:44.012559 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:58:44.015244 (+ 2685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:44.015292 (+ 48us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:44.015310 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:44.015655 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:44.016040 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:58:44.016043 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:58:44.016045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:58:44.016225 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:58:44.016801 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:58:44.016813 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:58:44.017214 (+ 401us) client_negotiation.cc:769] Sending connection context
0929 17:58:44.017431 (+ 217us) client_negotiation.cc:240] Negotiation successful
0929 17:58:44.017841 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":139,"threads_started":1}
I20260929 17:58:44.020246 18350 heartbeater.cc:347] Connected to a master server at 127.7.118.62:37689
I20260929 17:58:44.020525 18350 heartbeater.cc:464] Registering TS with master...
I20260929 17:58:44.021080 18350 heartbeater.cc:511] Master 127.7.118.62:37689 requested a full tablet report, sending...
I20260929 17:58:44.022176 17891 ts_manager.cc:194] Registered new tserver with Master: 281995707f1f453a85dfbb6e5901d6d9 (127.7.118.3:36569)
I20260929 17:58:44.022857 17891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.3@KRBTEST.COM'} at 127.7.118.3:38187
I20260929 17:58:44.030534 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:58:44.041428 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:44.032771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60380 (local address 127.7.118.62:37689)
0929 17:58:44.032970 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:44.032974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:44.032993 (+ 19us) server_negotiation.cc:407] Connection header received
0929 17:58:44.033160 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:44.033164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:44.033213 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:44.033305 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:58:44.034129 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.034780 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.035703 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.035956 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.036049 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:44.037550 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:58:44.037553 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:58:44.037555 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:58:44.037584 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:58:44.039244 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:44.039822 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:44.039826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:44.039828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:44.039931 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:58:44.040286 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:58:44.040289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:58:44.040290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:58:44.040660 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:58:44.040810 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:44.041164 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 17:58:44.041291 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260929 17:58:44.045432 17891 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:58:44.048846 17891 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:58:44.066481 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:44.059030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:36569 (local address 127.0.0.1:56874)
0929 17:58:44.060196 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:44.060272 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:44.060431 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:44.061109 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:44.061116 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:44.061148 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:44.061323 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:44.061331 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.062832 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.062836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:44.064280 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:44.064290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.065816 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.065820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:44.065950 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:44.065990 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 17:58:44.066118 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 17:58:44.066306 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":3,"thread_start_us":438,"threads_started":1}
I20260929 17:58:44.066493 18362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:44.059280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56874 (local address 127.7.118.3:36569)
0929 17:58:44.060043 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:44.060056 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:44.060358 (+ 302us) server_negotiation.cc:407] Connection header received
0929 17:58:44.060647 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:44.060655 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:44.060923 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:44.061097 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:44.061449 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.062568 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.064423 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.065663 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.065815 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:44.065993 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:44.066167 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 17:58:44.066311 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":187,"threads_started":1}
I20260929 17:58:44.066800 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:44.060702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:39545 (local address 127.0.0.1:51020)
0929 17:58:44.061008 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:44.061023 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:44.061095 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:44.062035 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:44.062038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:44.062055 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:44.062189 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:44.062196 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.063902 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.063906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:44.065115 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:44.065124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.066384 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.066389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:44.066488 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:44.066526 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 17:58:44.066578 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 17:58:44.066650 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":38,"thread_start_us":146,"threads_started":1}
I20260929 17:58:44.067708 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:44.061069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51020 (local address 127.7.118.2:39545)
0929 17:58:44.061425 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:44.061436 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:44.061457 (+ 21us) server_negotiation.cc:407] Connection header received
0929 17:58:44.061530 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:44.061535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:44.061738 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:44.061879 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:44.062369 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.063720 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.065255 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.066202 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.066930 (+ 728us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:44.067381 (+ 451us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:44.067442 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 17:58:44.067550 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":147,"threads_started":1}
I20260929 17:58:44.067998 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:44.060312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:36551 (local address 127.0.0.1:56734)
0929 17:58:44.060587 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:44.060601 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:44.060675 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:44.061886 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:44.061891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:44.061913 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:44.062015 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:44.062022 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.063756 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.063760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:44.065352 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:44.065364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.067543 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.067547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:44.067645 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:44.067669 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 17:58:44.067783 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 17:58:44.067859 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":104,"thread_start_us":124,"threads_started":1}
I20260929 17:58:44.068405 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:44.060499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56734 (local address 127.7.118.1:36551)
0929 17:58:44.061426 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:44.061434 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:44.061457 (+ 23us) server_negotiation.cc:407] Connection header received
0929 17:58:44.061530 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:44.061535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:44.061730 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:44.061879 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:44.062138 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.063589 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.065508 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.067409 (+ 1901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.067523 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:44.067652 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:44.068191 (+ 539us) server_negotiation.cc:299] Negotiation successful
0929 17:58:44.068290 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822,"thread_start_us":109,"threads_started":1}
I20260929 17:58:44.070534 18150 tablet_service.cc:1467] Processing CreateTablet for tablet 5bd1aa9043114047a8e968708999dc0e (DEFAULT_TABLE table=test-table [id=2b81ce3370674d90b234cb79c8c9782b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:58:44.070534 18284 tablet_service.cc:1467] Processing CreateTablet for tablet 5bd1aa9043114047a8e968708999dc0e (DEFAULT_TABLE table=test-table [id=2b81ce3370674d90b234cb79c8c9782b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:58:44.072037 18284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bd1aa9043114047a8e968708999dc0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:44.072027 18150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bd1aa9043114047a8e968708999dc0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:44.073909 18016 tablet_service.cc:1467] Processing CreateTablet for tablet 5bd1aa9043114047a8e968708999dc0e (DEFAULT_TABLE table=test-table [id=2b81ce3370674d90b234cb79c8c9782b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:58:44.075071 18016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bd1aa9043114047a8e968708999dc0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:58:44.080619 18368 tablet_bootstrap.cc:492] T 5bd1aa9043114047a8e968708999dc0e P f74c96497d5a4587ac0d6e8e4ce6aba9: Bootstrap starting.
I20260929 17:58:44.081398 18367 tablet_bootstrap.cc:492] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9: Bootstrap starting.
I20260929 17:58:44.084343 18369 tablet_bootstrap.cc:492] T 5bd1aa9043114047a8e968708999dc0e P 897ead4f78b64054902d0ef9148c6eaa: Bootstrap starting.
I20260929 17:58:44.085041 18367 tablet_bootstrap.cc:654] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:44.085299 18368 tablet_bootstrap.cc:654] T 5bd1aa9043114047a8e968708999dc0e P f74c96497d5a4587ac0d6e8e4ce6aba9: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:44.085978 18367 log.cc:824] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:44.086166 18368 log.cc:824] T 5bd1aa9043114047a8e968708999dc0e P f74c96497d5a4587ac0d6e8e4ce6aba9: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:44.087275 18369 tablet_bootstrap.cc:654] T 5bd1aa9043114047a8e968708999dc0e P 897ead4f78b64054902d0ef9148c6eaa: Neither blocks nor log segments found. Creating new log.
I20260929 17:58:44.088313 18368 tablet_bootstrap.cc:492] T 5bd1aa9043114047a8e968708999dc0e P f74c96497d5a4587ac0d6e8e4ce6aba9: No bootstrap required, opened a new log
I20260929 17:58:44.088302 18369 log.cc:824] T 5bd1aa9043114047a8e968708999dc0e P 897ead4f78b64054902d0ef9148c6eaa: Log is configured to *not* fsync() on all Append() calls
I20260929 17:58:44.088557 18368 ts_tablet_manager.cc:1397] T 5bd1aa9043114047a8e968708999dc0e P f74c96497d5a4587ac0d6e8e4ce6aba9: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 17:58:44.088558 18367 tablet_bootstrap.cc:492] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9: No bootstrap required, opened a new log
I20260929 17:58:44.088812 18367 ts_tablet_manager.cc:1397] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 17:58:44.091444 18369 tablet_bootstrap.cc:492] T 5bd1aa9043114047a8e968708999dc0e P 897ead4f78b64054902d0ef9148c6eaa: No bootstrap required, opened a new log
I20260929 17:58:44.091704 18369 ts_tablet_manager.cc:1397] T 5bd1aa9043114047a8e968708999dc0e P 897ead4f78b64054902d0ef9148c6eaa: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 17:58:44.093133 18368 raft_consensus.cc:348] T 5bd1aa9043114047a8e968708999dc0e P f74c96497d5a4587ac0d6e8e4ce6aba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "281995707f1f453a85dfbb6e5901d6d9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36569 } } peers { permanent_uuid: "897ead4f78b64054902d0ef9148c6eaa" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 36551 } } peers { permanent_uuid: "f74c96497d5a4587ac0d6e8e4ce6aba9" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 39545 } }
I20260929 17:58:44.093472 18368 raft_consensus.cc:374] T 5bd1aa9043114047a8e968708999dc0e P f74c96497d5a4587ac0d6e8e4ce6aba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:44.093549 18368 raft_consensus.cc:729] T 5bd1aa9043114047a8e968708999dc0e P f74c96497d5a4587ac0d6e8e4ce6aba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74c96497d5a4587ac0d6e8e4ce6aba9, State: Initialized, Role: FOLLOWER
I20260929 17:58:44.093317 18367 raft_consensus.cc:348] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "281995707f1f453a85dfbb6e5901d6d9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36569 } } peers { permanent_uuid: "897ead4f78b64054902d0ef9148c6eaa" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 36551 } } peers { permanent_uuid: "f74c96497d5a4587ac0d6e8e4ce6aba9" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 39545 } }
I20260929 17:58:44.093629 18367 raft_consensus.cc:374] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:44.093735 18367 raft_consensus.cc:729] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 281995707f1f453a85dfbb6e5901d6d9, State: Initialized, Role: FOLLOWER
I20260929 17:58:44.094113 18368 consensus_queue.cc:260] T 5bd1aa9043114047a8e968708999dc0e P f74c96497d5a4587ac0d6e8e4ce6aba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "281995707f1f453a85dfbb6e5901d6d9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36569 } } peers { permanent_uuid: "897ead4f78b64054902d0ef9148c6eaa" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 36551 } } peers { permanent_uuid: "f74c96497d5a4587ac0d6e8e4ce6aba9" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 39545 } }
I20260929 17:58:44.094328 18367 consensus_queue.cc:260] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "281995707f1f453a85dfbb6e5901d6d9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36569 } } peers { permanent_uuid: "897ead4f78b64054902d0ef9148c6eaa" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 36551 } } peers { permanent_uuid: "f74c96497d5a4587ac0d6e8e4ce6aba9" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 39545 } }
I20260929 17:58:44.095176 18368 ts_tablet_manager.cc:1428] T 5bd1aa9043114047a8e968708999dc0e P f74c96497d5a4587ac0d6e8e4ce6aba9: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260929 17:58:44.095433 18367 ts_tablet_manager.cc:1428] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 17:58:44.095594 18216 heartbeater.cc:503] Master 127.7.118.62:37689 was elected leader, sending a full tablet report...
I20260929 17:58:44.095835 18350 heartbeater.cc:503] Master 127.7.118.62:37689 was elected leader, sending a full tablet report...
I20260929 17:58:44.097108 18369 raft_consensus.cc:348] T 5bd1aa9043114047a8e968708999dc0e P 897ead4f78b64054902d0ef9148c6eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "281995707f1f453a85dfbb6e5901d6d9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36569 } } peers { permanent_uuid: "897ead4f78b64054902d0ef9148c6eaa" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 36551 } } peers { permanent_uuid: "f74c96497d5a4587ac0d6e8e4ce6aba9" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 39545 } }
I20260929 17:58:44.097455 18369 raft_consensus.cc:374] T 5bd1aa9043114047a8e968708999dc0e P 897ead4f78b64054902d0ef9148c6eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:58:44.097522 18369 raft_consensus.cc:729] T 5bd1aa9043114047a8e968708999dc0e P 897ead4f78b64054902d0ef9148c6eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 897ead4f78b64054902d0ef9148c6eaa, State: Initialized, Role: FOLLOWER
I20260929 17:58:44.098039 18369 consensus_queue.cc:260] T 5bd1aa9043114047a8e968708999dc0e P 897ead4f78b64054902d0ef9148c6eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "281995707f1f453a85dfbb6e5901d6d9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36569 } } peers { permanent_uuid: "897ead4f78b64054902d0ef9148c6eaa" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 36551 } } peers { permanent_uuid: "f74c96497d5a4587ac0d6e8e4ce6aba9" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 39545 } }
I20260929 17:58:44.098922 18082 heartbeater.cc:503] Master 127.7.118.62:37689 was elected leader, sending a full tablet report...
I20260929 17:58:44.099519 18369 ts_tablet_manager.cc:1428] T 5bd1aa9043114047a8e968708999dc0e P 897ead4f78b64054902d0ef9148c6eaa: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260929 17:58:44.177357 18374 raft_consensus.cc:482] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:58:44.177621 18374 raft_consensus.cc:504] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "281995707f1f453a85dfbb6e5901d6d9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36569 } } peers { permanent_uuid: "897ead4f78b64054902d0ef9148c6eaa" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 36551 } } peers { permanent_uuid: "f74c96497d5a4587ac0d6e8e4ce6aba9" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 39545 } }
I20260929 17:58:44.179067 18374 leader_election.cc:288] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 897ead4f78b64054902d0ef9148c6eaa (127.7.118.1:36551), f74c96497d5a4587ac0d6e8e4ce6aba9 (127.7.118.2:39545)
I20260929 17:58:44.183986 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:44.179491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:42669 (local address 127.7.118.1:36551)
0929 17:58:44.179628 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:44.179633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:44.179693 (+ 60us) server_negotiation.cc:407] Connection header received
0929 17:58:44.179766 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:44.179769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:44.179821 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:44.179926 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:44.180410 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.181243 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.182829 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.183319 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.183464 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:44.183538 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:44.183753 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 17:58:44.183851 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 17:58:44.183986 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:44.179337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:36551 (local address 127.7.118.3:42669)
0929 17:58:44.179471 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:44.179490 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:44.179601 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:44.180088 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:44.180091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:44.180109 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:44.180227 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:44.180234 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.181380 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.181383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:44.182685 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:44.182696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.183467 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.183471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:44.183594 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:44.183620 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 17:58:44.183709 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 17:58:44.183819 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":28}
I20260929 17:58:44.184859 18036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd1aa9043114047a8e968708999dc0e" candidate_uuid: "281995707f1f453a85dfbb6e5901d6d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "897ead4f78b64054902d0ef9148c6eaa" is_pre_election: true
I20260929 17:58:44.185163 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:44.179917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:51271 (local address 127.7.118.2:39545)
0929 17:58:44.180082 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:44.180087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:44.180441 (+ 354us) server_negotiation.cc:407] Connection header received
0929 17:58:44.180645 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:44.180648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:44.180712 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:44.180801 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:58:44.181176 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.182050 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.183632 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.184365 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.184474 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:44.184544 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:44.184941 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 17:58:44.185037 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 17:58:44.185248 18036 raft_consensus.cc:2454] T 5bd1aa9043114047a8e968708999dc0e P 897ead4f78b64054902d0ef9148c6eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 281995707f1f453a85dfbb6e5901d6d9 in term 0.
I20260929 17:58:44.185636 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:44.179678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:39545 (local address 127.7.118.3:51271)
0929 17:58:44.180234 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:44.180251 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:58:44.180520 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:58:44.180821 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:58:44.180823 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:58:44.180839 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:58:44.180981 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:44.180988 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.182201 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.182204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:44.183508 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:58:44.183520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.184661 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.184666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:58:44.184771 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:44.184796 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 17:58:44.185366 (+ 570us) client_negotiation.cc:240] Negotiation successful
0929 17:58:44.185454 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":1,"thread_start_us":121,"threads_started":1}
I20260929 17:58:44.185802 18239 leader_election.cc:302] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 281995707f1f453a85dfbb6e5901d6d9, 897ead4f78b64054902d0ef9148c6eaa; no voters:
I20260929 17:58:44.186172 18374 raft_consensus.cc:2781] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:58:44.186357 18374 raft_consensus.cc:482] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:58:44.186444 18374 raft_consensus.cc:3037] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:44.186722 18170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd1aa9043114047a8e968708999dc0e" candidate_uuid: "281995707f1f453a85dfbb6e5901d6d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74c96497d5a4587ac0d6e8e4ce6aba9" is_pre_election: true
I20260929 17:58:44.187032 18170 raft_consensus.cc:2454] T 5bd1aa9043114047a8e968708999dc0e P f74c96497d5a4587ac0d6e8e4ce6aba9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 281995707f1f453a85dfbb6e5901d6d9 in term 0.
I20260929 17:58:44.187934 18374 raft_consensus.cc:504] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "281995707f1f453a85dfbb6e5901d6d9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36569 } } peers { permanent_uuid: "897ead4f78b64054902d0ef9148c6eaa" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 36551 } } peers { permanent_uuid: "f74c96497d5a4587ac0d6e8e4ce6aba9" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 39545 } }
I20260929 17:58:44.188369 18374 leader_election.cc:288] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9 [CANDIDATE]: Term 1 election: Requested vote from peers 897ead4f78b64054902d0ef9148c6eaa (127.7.118.1:36551), f74c96497d5a4587ac0d6e8e4ce6aba9 (127.7.118.2:39545)
I20260929 17:58:44.188800 18036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd1aa9043114047a8e968708999dc0e" candidate_uuid: "281995707f1f453a85dfbb6e5901d6d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "897ead4f78b64054902d0ef9148c6eaa"
I20260929 17:58:44.188800 18170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd1aa9043114047a8e968708999dc0e" candidate_uuid: "281995707f1f453a85dfbb6e5901d6d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74c96497d5a4587ac0d6e8e4ce6aba9"
I20260929 17:58:44.188952 18170 raft_consensus.cc:3037] T 5bd1aa9043114047a8e968708999dc0e P f74c96497d5a4587ac0d6e8e4ce6aba9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:44.188964 18036 raft_consensus.cc:3037] T 5bd1aa9043114047a8e968708999dc0e P 897ead4f78b64054902d0ef9148c6eaa [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:58:44.190236 18036 raft_consensus.cc:2454] T 5bd1aa9043114047a8e968708999dc0e P 897ead4f78b64054902d0ef9148c6eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 281995707f1f453a85dfbb6e5901d6d9 in term 1.
I20260929 17:58:44.190373 18170 raft_consensus.cc:2454] T 5bd1aa9043114047a8e968708999dc0e P f74c96497d5a4587ac0d6e8e4ce6aba9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 281995707f1f453a85dfbb6e5901d6d9 in term 1.
I20260929 17:58:44.190716 18239 leader_election.cc:302] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 281995707f1f453a85dfbb6e5901d6d9, f74c96497d5a4587ac0d6e8e4ce6aba9; no voters:
I20260929 17:58:44.191152 18374 raft_consensus.cc:2781] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:58:44.191450 18374 raft_consensus.cc:686] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9 [term 1 LEADER]: Becoming Leader. State: Replica: 281995707f1f453a85dfbb6e5901d6d9, State: Running, Role: LEADER
I20260929 17:58:44.191848 18374 consensus_queue.cc:237] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "281995707f1f453a85dfbb6e5901d6d9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36569 } } peers { permanent_uuid: "897ead4f78b64054902d0ef9148c6eaa" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 36551 } } peers { permanent_uuid: "f74c96497d5a4587ac0d6e8e4ce6aba9" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 39545 } }
I20260929 17:58:44.195737 17888 catalog_manager.cc:5495] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 281995707f1f453a85dfbb6e5901d6d9 (127.7.118.3). New cstate: current_term: 1 leader_uuid: "281995707f1f453a85dfbb6e5901d6d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "281995707f1f453a85dfbb6e5901d6d9" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "897ead4f78b64054902d0ef9148c6eaa" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74c96497d5a4587ac0d6e8e4ce6aba9" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 39545 } health_report { overall_health: UNKNOWN } } }
I20260929 17:58:44.225781 18362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:44.221937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56880 (local address 127.7.118.3:36569)
0929 17:58:44.222087 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 17:58:44.222091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:58:44.222106 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:58:44.222319 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:58:44.222323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:58:44.222399 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:58:44.222513 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:58:44.222785 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.223306 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.224241 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:58:44.224489 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:58:44.224585 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:58:44.224938 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:58:44.225400 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:58:44.225499 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 17:58:44.225608 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 17:58:44.225664 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 17:58:44.232673 18170 raft_consensus.cc:1261] T 5bd1aa9043114047a8e968708999dc0e P f74c96497d5a4587ac0d6e8e4ce6aba9 [term 1 FOLLOWER]: Refusing update from remote peer 281995707f1f453a85dfbb6e5901d6d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:58:44.232673 18036 raft_consensus.cc:1261] T 5bd1aa9043114047a8e968708999dc0e P 897ead4f78b64054902d0ef9148c6eaa [term 1 FOLLOWER]: Refusing update from remote peer 281995707f1f453a85dfbb6e5901d6d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:58:44.233580 18374 consensus_queue.cc:1035] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "897ead4f78b64054902d0ef9148c6eaa" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:58:44.233861 18377 consensus_queue.cc:1035] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74c96497d5a4587ac0d6e8e4ce6aba9" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:58:44.243034 18381 mvcc.cc:205] Tried to move back new op lower bound from 7334726550448615424 to 7334726550293413888. Current Snapshot: MvccSnapshot[applied={T|T < 7334726550448615424}]
I20260929 17:58:44.243624 18382 mvcc.cc:205] Tried to move back new op lower bound from 7334726550448615424 to 7334726550293413888. Current Snapshot: MvccSnapshot[applied={T|T < 7334726550448615424}]
I20260929 17:58:44.255259 17889 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:58:44.255518 17889 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:58:44.258531 17889 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c410f626dd834ae08423d1549c677176: Sending DeleteTablet for 3 replicas of tablet 5bd1aa9043114047a8e968708999dc0e
I20260929 17:58:44.259673 18016 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bd1aa9043114047a8e968708999dc0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:58:44 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:56734
I20260929 17:58:44.259822 18150 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bd1aa9043114047a8e968708999dc0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:58:44 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:51020
I20260929 17:58:44.260291 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 17953
I20260929 17:58:44.260478 18391 tablet_replica.cc:330] T 5bd1aa9043114047a8e968708999dc0e P f74c96497d5a4587ac0d6e8e4ce6aba9: stopping tablet replica
I20260929 17:58:44.260439 18284 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bd1aa9043114047a8e968708999dc0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:58:44 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:56874
I20260929 17:58:44.260859 18391 raft_consensus.cc:2229] T 5bd1aa9043114047a8e968708999dc0e P f74c96497d5a4587ac0d6e8e4ce6aba9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:58:44.261238 18391 raft_consensus.cc:2258] T 5bd1aa9043114047a8e968708999dc0e P f74c96497d5a4587ac0d6e8e4ce6aba9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:58:44.264180 18391 ts_tablet_manager.cc:1905] T 5bd1aa9043114047a8e968708999dc0e P f74c96497d5a4587ac0d6e8e4ce6aba9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:58:44.268016 18391 ts_tablet_manager.cc:1918] T 5bd1aa9043114047a8e968708999dc0e P f74c96497d5a4587ac0d6e8e4ce6aba9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:58:44.268158 18391 log.cc:1196] T 5bd1aa9043114047a8e968708999dc0e P f74c96497d5a4587ac0d6e8e4ce6aba9: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-1/wal/wals/5bd1aa9043114047a8e968708999dc0e
I20260929 17:58:44.268529 18391 ts_tablet_manager.cc:1939] T 5bd1aa9043114047a8e968708999dc0e P f74c96497d5a4587ac0d6e8e4ce6aba9: Deleting consensus metadata
I20260929 17:58:44.268611 18392 tablet_replica.cc:330] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9: stopping tablet replica
W20260929 17:58:44.268674 17877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.118.1:36551 (error 108)
I20260929 17:58:44.268961 18392 raft_consensus.cc:2229] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:58:44.269402 18392 raft_consensus.cc:2258] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:58:44.269759 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 18087
I20260929 17:58:44.270885 18392 ts_tablet_manager.cc:1905] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:58:44.273756 18392 ts_tablet_manager.cc:1918] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:58:44.273890 18392 log.cc:1196] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704657566865-7640-0/minicluster-data/ts-2/wal/wals/5bd1aa9043114047a8e968708999dc0e
I20260929 17:58:44.274281 18392 ts_tablet_manager.cc:1939] T 5bd1aa9043114047a8e968708999dc0e P 281995707f1f453a85dfbb6e5901d6d9: Deleting consensus metadata
I20260929 17:58:44.276129 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:44.275677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:36551 (local address 127.0.0.1:56740)
0929 17:58:44.275921 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:44.276012 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.1:36551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260929 17:58:44.276346 17877 catalog_manager.cc:4584] TS 897ead4f78b64054902d0ef9148c6eaa (127.7.118.1:36551): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5bd1aa9043114047a8e968708999dc0e: Network error: Client connection negotiation failed: client connection to 127.7.118.1:36551: connect: Connection refused (error 111)
I20260929 17:58:44.276609 17877 catalog_manager.cc:4841] TS 281995707f1f453a85dfbb6e5901d6d9 (127.7.118.3:36569): tablet 5bd1aa9043114047a8e968708999dc0e (table test-table [id=2b81ce3370674d90b234cb79c8c9782b]) successfully deleted
I20260929 17:58:44.276772 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 18221
I20260929 17:58:44.277853 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:58:44.277589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:39545 (local address 127.0.0.1:51036)
0929 17:58:44.277713 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 17:58:44.277780 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.2:39545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 17:58:44.282641 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 17859
2026-09-29T17:58:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2512 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:58:44 dist-test-slave-r7ph krb5kdc[18396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:58:44 dist-test-slave-r7ph krb5kdc[18396](info): set up 2 sockets
Sep 29 17:58:44 dist-test-slave-r7ph krb5kdc[18396](info): commencing operation
krb5kdc: starting...
W20260929 17:58:45.343681 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 17:58:45 dist-test-slave-r7ph krb5kdc[18396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:58:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:58:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.7.118.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.7.118.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.7.118.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/rangeradmin_127.7.118.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.7.118.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/rangeradmin_127.7.118.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.7.118.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.7.118.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.7.118.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/rangerlookup_127.7.118.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.7.118.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/rangerlookup_127.7.118.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.118.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.7.118.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.118.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/HTTP_127.7.118.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.118.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/HTTP_127.7.118.20@KRBTEST.COM.keytab.
I20260929 17:58:45.533946 7640 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task8_cVK5/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/postgres -l logfile start
2026-09-29 17:58:48.585 UTC [18426] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 17:58:48.585 UTC [18426] LOG: listening on IPv4 address "127.7.118.20", port 59495
2026-09-29 17:58:48.590 UTC [18426] LOG: listening on Unix socket "/tmp/.s.PGSQL.59495"
2026-09-29 17:58:48.631 UTC [18429] LOG: database system was shut down at 2026-09-29 17:58:46 UTC
2026-09-29 17:58:48.636 UTC [18426] LOG: database system is ready to accept connections
I20260929 17:58:49.607120 7640 mini_postgres.cc:96] Postgres bound to 59495
2026-09-29 17:58:49.614 UTC [18437] FATAL: database "slave" does not exist
127.7.118.20:59495 - accepting connections
I20260929 17:58:49.615272 7640 mini_ranger.cc:161] Starting Ranger...
I20260929 17:58:49.631487 7640 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 17:58:50.271166 7640 mini_ranger.cc:87] Created ranger Postgres database
I20260929 17:58:50.271279 7640 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ranger-admin
2026-09-29 17:58:50,574 [I] DB FLAVOR :POSTGRES
2026-09-29 17:58:50,575 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 17:58:50,575 [I] Checking connection..
2026-09-29 17:58:50,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 17:58:50,844 [I] Checking connection passed.
2026-09-29 17:58:50,844 [I] --------- Verifying version history table ---------
2026-09-29 17:58:50,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 17:58:51,101 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 17:58:51,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 17:58:51,376 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 17:58:51,376 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 17:58:51,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 17:58:51.628 UTC [18501] WARNING: there is no transaction in progress
2026-09-29 17:58:51,635 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 17:58:51,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 17:58:51,899 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 17:58:51,899 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 17:58:51,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 17:58:52,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 17:58:52,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-r7ph.c.gcp-upstream.internal','N') ;"
2026-09-29 17:58:52,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 17:58:52,996 [I] Table x_portal_user does not exist in database ranger
2026-09-29 17:58:52,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 17:58:53,268 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 17:58:53,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 17:58:53,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 17:58:53,825 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 17:58:53,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 17:58:55.066 UTC [18609] WARNING: there is no transaction in progress
2026-09-29 17:58:55.088 UTC [18609] WARNING: there is no transaction in progress
2026-09-29 17:58:55.109 UTC [18609] WARNING: there is no transaction in progress
2026-09-29 17:58:55.130 UTC [18609] WARNING: there is no transaction in progress
2026-09-29 17:58:55.153 UTC [18609] WARNING: there is no transaction in progress
2026-09-29 17:58:55.175 UTC [18609] WARNING: there is no transaction in progress
2026-09-29 17:58:55.242 UTC [18609] WARNING: there is no transaction in progress
2026-09-29 17:58:55.250 UTC [18609] WARNING: there is no transaction in progress
2026-09-29 17:58:55.268 UTC [18609] WARNING: there is no transaction in progress
2026-09-29 17:58:55.281 UTC [18609] WARNING: there is no transaction in progress
2026-09-29 17:58:55.293 UTC [18609] WARNING: there is no transaction in progress
2026-09-29 17:58:55.309 UTC [18609] WARNING: there is no transaction in progress
2026-09-29 17:58:55.321 UTC [18609] WARNING: there is no transaction in progress
2026-09-29 17:58:55.329 UTC [18609] WARNING: there is no transaction in progress
2026-09-29 17:58:55.339 UTC [18609] WARNING: there is no transaction in progress
2026-09-29 17:58:55.347 UTC [18609] WARNING: there is no transaction in progress
2026-09-29 17:58:56,232 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 17:58:56,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 17:58:56,472 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 17:58:56,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 17:58:56,752 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 17:58:56,752 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 17:58:56,998 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 17:58:56,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 17:58:57,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 17:58:57,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-r7ph.c.gcp-upstream.internal';"
2026-09-29 17:58:57,812 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 17:58:57,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 17:58:58,087 [I] DB_PATCHES have already been applied
I20260929 17:58:58.093104 7640 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ranger-admin:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task8_cVK5/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 17:58:58.093190 7640 mini_ranger.cc:193] Using host: 127.7.118.20
I20260929 17:58:58.097234 7640 mini_ranger.cc:227] Ranger admin URL: http://127.7.118.20:45857
Sep 29, 2026 5:58:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 5:58:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 5:58:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 5:58:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 5:58:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 5:58:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 5:58:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 17:58:59.261240 7640 mini_ranger.cc:160] Time spent starting Ranger: real 9.646s user 0.000s sys 0.012s
Sep 29, 2026 5:59:21 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 5:59:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 5:59:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 5:59:21 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 5:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 5:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 5:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 5:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 5:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 5:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 5:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 5:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 5:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 5:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 5:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 5:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 5:59:21 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 5:59:22 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 17:59:22.828591 7640 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.7.118.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.7.118.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.7.118.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/rangerkms_127.7.118.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.7.118.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/rangerkms_127.7.118.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.118.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.7.118.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.118.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/HTTP_127.7.118.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.118.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/HTTP_127.7.118.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 17:59:22 dist-test-slave-r7ph krb5kdc[18396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704762, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 17:59:22.923296 7640 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 17:59:22.935120 7640 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 17:59:24.158530 7640 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 17:59:24.158653 7640 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ranger-kms
I20260929 17:59:24.158766 7640 mini_ranger_kms.cc:227] Using postgres at 127.7.118.20:59495
2026-09-29 17:59:24,365 [I] DB FLAVOR :POSTGRES
2026-09-29 17:59:24,366 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 17:59:24,366 [I] Checking connection
2026-09-29 17:59:24,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:59495/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 17:59:24,647 [I] connection success
2026-09-29 17:59:24,647 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 17:59:24,647 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 17:59:24,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:59495/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 17:59:24,942 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 17:59:24,942 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 17:59:24,943 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 17:59:24,943 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:59495/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 17:59:25,261 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 17:59:25.828585 7640 mini_ranger_kms.cc:310] Created kms service
I20260929 17:59:26.051689 7640 mini_ranger_kms.cc:326] Created kudu user
I20260929 17:59:26.106693 7640 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 17:59:26 dist-test-slave-r7ph krb5kdc[18396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704762, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.7.118.20@KRBTEST.COM
I20260929 17:59:26.535980 7640 mini_ranger_kms.cc:384] Added ranger policy
I20260929 17:59:26.536140 7640 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ranger-kms:/tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 17:59:26.536170 7640 mini_ranger_kms.cc:240] Using host: 127.7.118.20
I20260929 17:59:26.540398 7640 mini_ranger_kms.cc:276] Ranger KMS PID: 18790
I20260929 17:59:26.540632 7640 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.7.118.20:51329
Sep 29, 2026 5:59:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 5:59:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 5:59:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 5:59:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 5:59:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 5:59:28 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 17:59:28.902484 7640 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.979s user 0.000s sys 0.189s
I20260929 17:59:28.902612 7640 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 17:59:28.902721 7640 mini_ranger_kms.cc:401] 127.7.118.20:51329/kms/v1/keys
Sep 29 17:59:28 dist-test-slave-r7ph krb5kdc[18396](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790704768, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.118.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 17:59:29 dist-test-slave-r7ph krb5kdc[18396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704768, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.118.20@KRBTEST.COM for HTTP/127.7.118.20@KRBTEST.COM
Sep 29 17:59:29 dist-test-slave-r7ph krb5kdc[18396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704768, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.118.20@KRBTEST.COM for HTTP/127.7.118.20@KRBTEST.COM
Sep 29 17:59:32 dist-test-slave-r7ph krb5kdc[18396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 17:59:32.363332 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:43963
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:43963
--ranger_config_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.118.20:51329/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:59:32.502204 18840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:59:32.502557 18840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:59:32.502599 18840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:59:32.507267 18840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:59:32.507361 18840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:59:32.507382 18840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:59:32.507400 18840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:59:32.507416 18840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:59:32.514328 18840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/master-0/wal
--ranger_kms_url=127.7.118.20:51329/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:43963
--ranger_config_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:43963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:59:32.515476 18840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:59:32.516505 18840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:59:32.528636 18846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:59:32.528741 18840 server_base.cc:1034] running on GCE node
W20260929 17:59:32.528844 18845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:59:32.528640 18848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:59:32.529625 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:59:32.531066 18840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:59:32.532310 18840 hybrid_clock.cc:648] HybridClock initialized: now 1790704772532281 us; error 49 us; skew 500 ppm
Sep 29 17:59:32 dist-test-slave-r7ph krb5kdc[18396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704772, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:59:32.545053 18840 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 17:59:32.546332 18840 webserver.cc:460] Webserver started at http://127.7.118.62:46357/ using document root <none> and password file <none>
I20260929 17:59:32.547094 18840 fs_manager.cc:359] Metadata directory not provided
I20260929 17:59:32.547194 18840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:59:32.547446 18840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 17:59:32 dist-test-slave-r7ph krb5kdc[18396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704772, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for HTTP/127.7.118.20@KRBTEST.COM
I20260929 17:59:32.616210 18840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "3ccf9e85585d4673b8eaca6ed69d8331"
format_stamp: "Formatted at 2026-09-29 17:59:32 on dist-test-slave-r7ph"
server_key: "08d06c041acd1b75b3770d2a437b2096"
server_key_iv: "6c3715b169b5ea07067fcee762a3a8c7"
server_key_version: "kuduclusterkey@0"
I20260929 17:59:32.617085 18840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "3ccf9e85585d4673b8eaca6ed69d8331"
format_stamp: "Formatted at 2026-09-29 17:59:32 on dist-test-slave-r7ph"
server_key: "08d06c041acd1b75b3770d2a437b2096"
server_key_iv: "6c3715b169b5ea07067fcee762a3a8c7"
server_key_version: "kuduclusterkey@0"
I20260929 17:59:32.621428 18840 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 17:59:32.641789 18855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:59:32.643798 18840 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 17:59:32.643971 18840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "3ccf9e85585d4673b8eaca6ed69d8331"
format_stamp: "Formatted at 2026-09-29 17:59:32 on dist-test-slave-r7ph"
server_key: "08d06c041acd1b75b3770d2a437b2096"
server_key_iv: "6c3715b169b5ea07067fcee762a3a8c7"
server_key_version: "kuduclusterkey@0"
I20260929 17:59:32.644124 18840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:59:32.661347 18840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:59:32.665272 18840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:59:32.665556 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:59:32.674539 18840 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:43963
I20260929 17:59:32.674559 18907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:43963 every 8 connection(s)
I20260929 17:59:32.675629 18840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 17:59:32.679463 18908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:59:32.684265 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 18840
I20260929 17:59:32.684448 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 17:59:32.684770 7640 external_mini_cluster.cc:1428] Setting key 22fa462e30e7315f995d270069510abc
I20260929 17:59:32.687155 18908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ccf9e85585d4673b8eaca6ed69d8331: Bootstrap starting.
I20260929 17:59:32.690117 18908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ccf9e85585d4673b8eaca6ed69d8331: Neither blocks nor log segments found. Creating new log.
I20260929 17:59:32.691154 18908 log.cc:824] T 00000000000000000000000000000000 P 3ccf9e85585d4673b8eaca6ed69d8331: Log is configured to *not* fsync() on all Append() calls
I20260929 17:59:32.694042 18908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ccf9e85585d4673b8eaca6ed69d8331: No bootstrap required, opened a new log
Sep 29 17:59:32 dist-test-slave-r7ph krb5kdc[18396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:59:32.698676 18908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ccf9e85585d4673b8eaca6ed69d8331 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ccf9e85585d4673b8eaca6ed69d8331" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 43963 } }
I20260929 17:59:32.699045 18908 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ccf9e85585d4673b8eaca6ed69d8331 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:59:32.699162 18908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ccf9e85585d4673b8eaca6ed69d8331 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccf9e85585d4673b8eaca6ed69d8331, State: Initialized, Role: FOLLOWER
I20260929 17:59:32.699743 18908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ccf9e85585d4673b8eaca6ed69d8331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ccf9e85585d4673b8eaca6ed69d8331" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 43963 } }
I20260929 17:59:32.699954 18908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ccf9e85585d4673b8eaca6ed69d8331 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:59:32.700053 18908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ccf9e85585d4673b8eaca6ed69d8331 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:59:32.700198 18908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ccf9e85585d4673b8eaca6ed69d8331 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:59:32.701035 18911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:59:32.686542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43734 (local address 127.7.118.62:43963)
0929 17:59:32.686995 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0929 17:59:32.687018 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 17:59:32.687070 (+ 52us) server_negotiation.cc:407] Connection header received
0929 17:59:32.687789 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:59:32.687809 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:59:32.688195 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:59:32.688577 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:59:32.689418 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:59:32.690549 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:59:32.691538 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:59:32.692277 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:59:32.692465 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:59:32.695769 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:59:32.695777 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:59:32.695797 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:59:32.695833 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:59:32.697623 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:59:32.698187 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:59:32.698193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:59:32.698198 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:59:32.698413 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:59:32.698732 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:59:32.698735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:59:32.698737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:59:32.699304 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:59:32.699543 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 17:59:32.699834 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 17:59:32.700165 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":155,"threads_started":1}
I20260929 17:59:32.701328 18908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ccf9e85585d4673b8eaca6ed69d8331 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ccf9e85585d4673b8eaca6ed69d8331" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 43963 } }
I20260929 17:59:32.701707 18908 leader_election.cc:302] T 00000000000000000000000000000000 P 3ccf9e85585d4673b8eaca6ed69d8331 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ccf9e85585d4673b8eaca6ed69d8331; no voters:
I20260929 17:59:32.702078 18908 leader_election.cc:288] T 00000000000000000000000000000000 P 3ccf9e85585d4673b8eaca6ed69d8331 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:59:32.702224 18913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ccf9e85585d4673b8eaca6ed69d8331 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:59:32.702545 18913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ccf9e85585d4673b8eaca6ed69d8331 [term 1 LEADER]: Becoming Leader. State: Replica: 3ccf9e85585d4673b8eaca6ed69d8331, State: Running, Role: LEADER
I20260929 17:59:32.702917 18913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ccf9e85585d4673b8eaca6ed69d8331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ccf9e85585d4673b8eaca6ed69d8331" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 43963 } }
I20260929 17:59:32.703428 18908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ccf9e85585d4673b8eaca6ed69d8331 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:59:32.704607 18914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ccf9e85585d4673b8eaca6ed69d8331 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ccf9e85585d4673b8eaca6ed69d8331. Latest consensus state: current_term: 1 leader_uuid: "3ccf9e85585d4673b8eaca6ed69d8331" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ccf9e85585d4673b8eaca6ed69d8331" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 43963 } } }
I20260929 17:59:32.704751 18914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ccf9e85585d4673b8eaca6ed69d8331 [sys.catalog]: This master's current role is: LEADER
I20260929 17:59:32.705351 18922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:59:32.705612 18915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ccf9e85585d4673b8eaca6ed69d8331 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ccf9e85585d4673b8eaca6ed69d8331" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ccf9e85585d4673b8eaca6ed69d8331" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 43963 } } }
I20260929 17:59:32.705717 18915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ccf9e85585d4673b8eaca6ed69d8331 [sys.catalog]: This master's current role is: LEADER
I20260929 17:59:32.708611 18922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:59:32.709304 18908 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 17:59:32.718652 18922 catalog_manager.cc:1347] Generated new cluster ID: ae4e9755298a4135870521b064580ce7
I20260929 17:59:32.718743 18922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:59:32.765929 18922 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:59:32.767668 18922 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:59:32.775966 18922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ccf9e85585d4673b8eaca6ed69d8331: Generated new TSK 0
I20260929 17:59:32.777034 18922 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.118.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
I20260929 17:59:33.546650 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.118.20:51329/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:0
--local_ip_for_outbound_sockets=127.7.118.1
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:43963
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:59:33.718681 18953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:59:33.719025 18953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:59:33.719098 18953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:59:33.724452 18953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:59:33.724606 18953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:59:33.724721 18953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 17:59:33.732678 18953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.7.118.20:51329/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:43963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:59:33.734050 18953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:59:33.735296 18953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:59:33.749425 18960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:59:33.749662 18953 server_base.cc:1034] running on GCE node
W20260929 17:59:33.749395 18961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:59:33.750010 18963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:59:33.750626 18953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:59:33.751441 18953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:59:33.752626 18953 hybrid_clock.cc:648] HybridClock initialized: now 1790704773752621 us; error 112 us; skew 500 ppm
Sep 29 17:59:33 dist-test-slave-r7ph krb5kdc[18396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704773, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:59:33.756283 18953 init.cc:377] Logged in from keytab as kudu/127.7.118.1@KRBTEST.COM (short username kudu)
I20260929 17:59:33.757632 18953 webserver.cc:460] Webserver started at http://127.7.118.1:42331/ using document root <none> and password file <none>
I20260929 17:59:33.758702 18953 fs_manager.cc:359] Metadata directory not provided
I20260929 17:59:33.758827 18953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:59:33.759099 18953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 17:59:33 dist-test-slave-r7ph krb5kdc[18396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704773, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for HTTP/127.7.118.20@KRBTEST.COM
I20260929 17:59:33.778093 18953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-0/data/instance:
uuid: "d2bbe34e12a24b8c90c8822acc7517b1"
format_stamp: "Formatted at 2026-09-29 17:59:33 on dist-test-slave-r7ph"
server_key: "7c36712ecff0a93b2de67b5a07456037"
server_key_iv: "025701c6f420023701b32cb66048a032"
server_key_version: "kuduclusterkey@0"
I20260929 17:59:33.779078 18953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance:
uuid: "d2bbe34e12a24b8c90c8822acc7517b1"
format_stamp: "Formatted at 2026-09-29 17:59:33 on dist-test-slave-r7ph"
server_key: "7c36712ecff0a93b2de67b5a07456037"
server_key_iv: "025701c6f420023701b32cb66048a032"
server_key_version: "kuduclusterkey@0"
I20260929 17:59:33.785028 18953 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 17:59:33.822875 18970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:59:33.831285 18953 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.005s sys 0.000s
I20260929 17:59:33.831477 18953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "d2bbe34e12a24b8c90c8822acc7517b1"
format_stamp: "Formatted at 2026-09-29 17:59:33 on dist-test-slave-r7ph"
server_key: "7c36712ecff0a93b2de67b5a07456037"
server_key_iv: "025701c6f420023701b32cb66048a032"
server_key_version: "kuduclusterkey@0"
I20260929 17:59:33.831667 18953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:59:33.885018 18953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:59:33.888530 18953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:59:33.888759 18953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:59:33.889444 18953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:59:33.890476 18953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:59:33.890550 18953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:59:33.890636 18953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:59:33.890697 18953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:59:33.903188 18953 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:39167
I20260929 17:59:33.903225 19085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:39167 every 8 connection(s)
I20260929 17:59:33.904477 18953 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 17:59:33.908774 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 18953
I20260929 17:59:33.908913 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance
I20260929 17:59:33.909222 7640 external_mini_cluster.cc:1428] Setting key 561c5b04e5da831107cc51702d6f4a1d
Sep 29 17:59:33 dist-test-slave-r7ph krb5kdc[18396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704773, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.118.2@KRBTEST.COM; defaulting to no policy
I20260929 17:59:33.924470 19089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:59:33.907536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:43963 (local address 127.7.118.1:34705)
0929 17:59:33.908032 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0929 17:59:33.908093 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 17:59:33.909241 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:59:33.910176 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:59:33.910187 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:59:33.910599 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:59:33.911332 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:59:33.911354 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:59:33.912656 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:59:33.912663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:59:33.913853 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:59:33.913863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:59:33.914776 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:59:33.914782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:59:33.915041 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:59:33.915832 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:59:33.915859 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:59:33.917928 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:59:33.920725 (+ 2797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:59:33.920735 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:59:33.920759 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:59:33.921222 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:59:33.921619 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:59:33.921625 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:59:33.921629 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:59:33.921860 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:59:33.922520 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:59:33.922530 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:59:33.922924 (+ 394us) client_negotiation.cc:769] Sending connection context
0929 17:59:33.923157 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 17:59:33.923553 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":145,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.2@KRBTEST.COM" created.
I20260929 17:59:33.925288 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:59:33.906993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:34705 (local address 127.7.118.62:43963)
0929 17:59:33.907271 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 17:59:33.907276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:59:33.908208 (+ 932us) server_negotiation.cc:407] Connection header received
0929 17:59:33.909533 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:59:33.909538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:59:33.909605 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:59:33.909681 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:59:33.911534 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:59:33.912452 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:59:33.913990 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:59:33.914437 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:59:33.914556 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:59:33.918230 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:59:33.918237 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:59:33.918276 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:59:33.918316 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:59:33.920535 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:59:33.921371 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:59:33.921375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:59:33.921377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:59:33.921481 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:59:33.921980 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:59:33.921983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:59:33.921985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:59:33.922295 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:59:33.922401 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 17:59:33.924931 (+ 2530us) server_negotiation.cc:299] Negotiation successful
0929 17:59:33.925112 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":97,"threads_started":1}
I20260929 17:59:33.927211 19086 heartbeater.cc:347] Connected to a master server at 127.7.118.62:43963
I20260929 17:59:33.927484 19086 heartbeater.cc:464] Registering TS with master...
I20260929 17:59:33.928138 19086 heartbeater.cc:511] Master 127.7.118.62:43963 requested a full tablet report, sending...
I20260929 17:59:33.930229 18872 ts_manager.cc:194] Registered new tserver with Master: d2bbe34e12a24b8c90c8822acc7517b1 (127.7.118.1:39167)
I20260929 17:59:33.931859 18872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.1@KRBTEST.COM'} at 127.7.118.1:34705
Sep 29 17:59:33 dist-test-slave-r7ph krb5kdc[18396](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790704773, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
I20260929 17:59:33.946050 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.118.20:51329/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:0
--local_ip_for_outbound_sockets=127.7.118.2
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:43963
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:59:34.117847 19092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:59:34.118312 19092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:59:34.118378 19092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:59:34.124910 19092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:59:34.125020 19092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:59:34.125109 19092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 17:59:34.133591 19092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.7.118.20:51329/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:43963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:59:34.135268 19092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:59:34.136740 19092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:59:34.150827 19101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:59:34.151247 19098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:59:34.151423 19099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:59:34.151638 19092 server_base.cc:1034] running on GCE node
I20260929 17:59:34.152195 19092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:59:34.153005 19092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:59:34.154178 19092 hybrid_clock.cc:648] HybridClock initialized: now 1790704774154131 us; error 61 us; skew 500 ppm
Sep 29 17:59:34 dist-test-slave-r7ph krb5kdc[18396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704774, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:59:34.157968 19092 init.cc:377] Logged in from keytab as kudu/127.7.118.2@KRBTEST.COM (short username kudu)
I20260929 17:59:34.159363 19092 webserver.cc:460] Webserver started at http://127.7.118.2:33103/ using document root <none> and password file <none>
I20260929 17:59:34.160188 19092 fs_manager.cc:359] Metadata directory not provided
I20260929 17:59:34.160287 19092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:59:34.160574 19092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 17:59:34 dist-test-slave-r7ph krb5kdc[18396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704774, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for HTTP/127.7.118.20@KRBTEST.COM
I20260929 17:59:34.187376 19092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-1/data/instance:
uuid: "4fcaa158682949c18a7841b9a36972b6"
format_stamp: "Formatted at 2026-09-29 17:59:34 on dist-test-slave-r7ph"
server_key: "5033e6dee554c882af0cd22d789ef062"
server_key_iv: "5a673cafca6e0e80e0b0a07f2b0a0f03"
server_key_version: "kuduclusterkey@0"
I20260929 17:59:34.188234 19092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance:
uuid: "4fcaa158682949c18a7841b9a36972b6"
format_stamp: "Formatted at 2026-09-29 17:59:34 on dist-test-slave-r7ph"
server_key: "5033e6dee554c882af0cd22d789ef062"
server_key_iv: "5a673cafca6e0e80e0b0a07f2b0a0f03"
server_key_version: "kuduclusterkey@0"
I20260929 17:59:34.192828 19092 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:59:34.212828 19108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:59:34.214993 19092 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 17:59:34.215199 19092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "4fcaa158682949c18a7841b9a36972b6"
format_stamp: "Formatted at 2026-09-29 17:59:34 on dist-test-slave-r7ph"
server_key: "5033e6dee554c882af0cd22d789ef062"
server_key_iv: "5a673cafca6e0e80e0b0a07f2b0a0f03"
server_key_version: "kuduclusterkey@0"
I20260929 17:59:34.215315 19092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:59:34.252842 19092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:59:34.261188 19092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:59:34.261494 19092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:59:34.262424 19092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:59:34.263650 19092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:59:34.263768 19092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:59:34.263870 19092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:59:34.263935 19092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:59:34.279646 19092 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:42007
I20260929 17:59:34.281061 19092 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 17:59:34.283147 19221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:42007 every 8 connection(s)
I20260929 17:59:34.286481 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 19092
I20260929 17:59:34.286619 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance
I20260929 17:59:34.286939 7640 external_mini_cluster.cc:1428] Setting key 7a19ccf4cf7ee2a88526f80752b4da48
Sep 29 17:59:34 dist-test-slave-r7ph krb5kdc[18396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704774, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:59:34.310861 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:59:34.285410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:40659 (local address 127.7.118.62:43963)
0929 17:59:34.285569 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 17:59:34.285575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:59:34.287255 (+ 1680us) server_negotiation.cc:407] Connection header received
0929 17:59:34.292038 (+ 4783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:59:34.292044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:59:34.292112 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:59:34.292172 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:59:34.294338 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:59:34.295115 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:59:34.297078 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:59:34.297431 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:59:34.297540 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:59:34.304313 (+ 6773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:59:34.304318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:59:34.304320 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:59:34.304360 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:59:34.306496 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:59:34.307349 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:59:34.307353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:59:34.307355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:59:34.307455 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:59:34.307935 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:59:34.307938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:59:34.307940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:59:34.308188 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:59:34.308273 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 17:59:34.310444 (+ 2171us) server_negotiation.cc:299] Negotiation successful
0929 17:59:34.310646 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 17:59:34.312824 19224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:59:34.285703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:43963 (local address 127.7.118.2:40659)
0929 17:59:34.287061 (+ 1358us) negotiation.cc:107] Waiting for socket to connect
0929 17:59:34.287130 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 17:59:34.291677 (+ 4547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:59:34.292774 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:59:34.292787 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:59:34.293236 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:59:34.294078 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:59:34.294103 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:59:34.295321 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:59:34.295324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:59:34.296897 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:59:34.296911 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:59:34.297807 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:59:34.297811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:59:34.298043 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:59:34.301812 (+ 3769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:59:34.301849 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:59:34.303987 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:59:34.306722 (+ 2735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:59:34.306730 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:59:34.306753 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:59:34.307210 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:59:34.307592 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:59:34.307595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:59:34.307596 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:59:34.307822 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:59:34.308349 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:59:34.308357 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:59:34.308776 (+ 419us) client_negotiation.cc:769] Sending connection context
0929 17:59:34.311134 (+ 2358us) client_negotiation.cc:240] Negotiation successful
0929 17:59:34.311586 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1119,"thread_start_us":155,"threads_started":1}
I20260929 17:59:34.314421 19222 heartbeater.cc:347] Connected to a master server at 127.7.118.62:43963
I20260929 17:59:34.314725 19222 heartbeater.cc:464] Registering TS with master...
I20260929 17:59:34.315371 19222 heartbeater.cc:511] Master 127.7.118.62:43963 requested a full tablet report, sending...
I20260929 17:59:34.316699 18872 ts_manager.cc:194] Registered new tserver with Master: 4fcaa158682949c18a7841b9a36972b6 (127.7.118.2:42007)
I20260929 17:59:34.317543 18872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.2@KRBTEST.COM'} at 127.7.118.2:40659
WARNING: no policy specified for kudu/127.7.118.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
I20260929 17:59:34.356345 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.118.20:51329/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:0
--local_ip_for_outbound_sockets=127.7.118.3
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:43963
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 17:59:34 dist-test-slave-r7ph krb5kdc[18396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704773, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for HTTP/127.7.118.20@KRBTEST.COM
W20260929 17:59:34.549995 19227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:59:34.550439 19227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:59:34.550561 19227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:59:34.557042 19227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:59:34.557204 19227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:59:34.557323 19227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 17:59:34.565629 19227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.7.118.20:51329/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:43963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:59:34.567303 19227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:59:34.568759 19227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:59:34.585670 19233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:59:34.585670 19235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:59:34.586037 19232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:59:34.587901 19227 server_base.cc:1034] running on GCE node
I20260929 17:59:34.588515 19227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:59:34.589393 19227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:59:34.604950 19227 hybrid_clock.cc:648] HybridClock initialized: now 1790704774604790 us; error 169 us; skew 500 ppm
Sep 29 17:59:34 dist-test-slave-r7ph krb5kdc[18396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704774, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:59:34.609355 19227 init.cc:377] Logged in from keytab as kudu/127.7.118.3@KRBTEST.COM (short username kudu)
I20260929 17:59:34.610685 19227 webserver.cc:460] Webserver started at http://127.7.118.3:42347/ using document root <none> and password file <none>
I20260929 17:59:34.611436 19227 fs_manager.cc:359] Metadata directory not provided
I20260929 17:59:34.611538 19227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:59:34.611788 19227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 17:59:34 dist-test-slave-r7ph krb5kdc[18396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704774, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for HTTP/127.7.118.20@KRBTEST.COM
I20260929 17:59:34.638870 19227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-2/data/instance:
uuid: "51134b442e0e4bb7abc492e42896d242"
format_stamp: "Formatted at 2026-09-29 17:59:34 on dist-test-slave-r7ph"
server_key: "369a2f54b2de49d31423c1f0ae4aade1"
server_key_iv: "be64aad3cbf96f637678db147084f6b6"
server_key_version: "kuduclusterkey@0"
I20260929 17:59:34.639782 19227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance:
uuid: "51134b442e0e4bb7abc492e42896d242"
format_stamp: "Formatted at 2026-09-29 17:59:34 on dist-test-slave-r7ph"
server_key: "369a2f54b2de49d31423c1f0ae4aade1"
server_key_iv: "be64aad3cbf96f637678db147084f6b6"
server_key_version: "kuduclusterkey@0"
I20260929 17:59:34.652917 19227 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.006s sys 0.000s
I20260929 17:59:34.695262 19242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:59:34.697321 19227 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 17:59:34.697649 19227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "51134b442e0e4bb7abc492e42896d242"
format_stamp: "Formatted at 2026-09-29 17:59:34 on dist-test-slave-r7ph"
server_key: "369a2f54b2de49d31423c1f0ae4aade1"
server_key_iv: "be64aad3cbf96f637678db147084f6b6"
server_key_version: "kuduclusterkey@0"
I20260929 17:59:34.697824 19227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:59:34.731984 19227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:59:34.753149 19227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:59:34.753554 19227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:59:34.754467 19227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:59:34.755761 19227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:59:34.755877 19227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:59:34.756001 19227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:59:34.756071 19227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:59:34.780439 19227 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:36127
I20260929 17:59:34.782159 19227 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 17:59:34.792526 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 19227
I20260929 17:59:34.792681 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance
I20260929 17:59:34.792994 7640 external_mini_cluster.cc:1428] Setting key 1cb0057e98f463f93e09ebda846087cb
I20260929 17:59:34.794499 19355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:36127 every 8 connection(s)
Sep 29 17:59:34 dist-test-slave-r7ph krb5kdc[18396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704774, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 17:59:34.841948 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:59:34.798926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:48767 (local address 127.7.118.62:43963)
0929 17:59:34.799056 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 17:59:34.799060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:59:34.803623 (+ 4563us) server_negotiation.cc:407] Connection header received
0929 17:59:34.810465 (+ 6842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:59:34.810470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:59:34.810547 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:59:34.810612 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:59:34.813029 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:59:34.814054 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:59:34.818881 (+ 4827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:59:34.819614 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:59:34.820044 (+ 430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:59:34.835620 (+ 15576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:59:34.835628 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:59:34.835630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:59:34.835674 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:59:34.838348 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:59:34.839266 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:59:34.839270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:59:34.839272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:59:34.839402 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:59:34.839892 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:59:34.839896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:59:34.839898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:59:34.840164 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:59:34.840258 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 17:59:34.841462 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0929 17:59:34.841627 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 17:59:34.844803 19358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:59:34.797584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:43963 (local address 127.7.118.3:48767)
0929 17:59:34.803371 (+ 5787us) negotiation.cc:107] Waiting for socket to connect
0929 17:59:34.803477 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0929 17:59:34.805172 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:59:34.811220 (+ 6048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:59:34.811232 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:59:34.811678 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:59:34.812734 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:59:34.812766 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:59:34.814314 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:59:34.814317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:59:34.816135 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:59:34.816152 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:59:34.820432 (+ 4280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:59:34.820438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:59:34.820757 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:59:34.821727 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:59:34.821753 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:59:34.835342 (+ 13589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:59:34.838609 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:59:34.838618 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:59:34.838642 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:59:34.839119 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:59:34.839536 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:59:34.839539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:59:34.839542 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:59:34.839771 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:59:34.840485 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:59:34.840515 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:59:34.841096 (+ 581us) client_negotiation.cc:769] Sending connection context
0929 17:59:34.842898 (+ 1802us) client_negotiation.cc:240] Negotiation successful
0929 17:59:34.843429 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5450,"thread_start_us":176,"threads_started":1}
I20260929 17:59:34.846990 19356 heartbeater.cc:347] Connected to a master server at 127.7.118.62:43963
I20260929 17:59:34.847295 19356 heartbeater.cc:464] Registering TS with master...
I20260929 17:59:34.847963 19356 heartbeater.cc:511] Master 127.7.118.62:43963 requested a full tablet report, sending...
I20260929 17:59:34.852308 18872 ts_manager.cc:194] Registered new tserver with Master: 51134b442e0e4bb7abc492e42896d242 (127.7.118.3:36127)
I20260929 17:59:34.853121 18872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.3@KRBTEST.COM'} at 127.7.118.3:48767
I20260929 17:59:34.859679 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:59:34.899816 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:59:34.874898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43746 (local address 127.7.118.62:43963)
0929 17:59:34.875058 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 17:59:34.875063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:59:34.882707 (+ 7644us) server_negotiation.cc:407] Connection header received
0929 17:59:34.882976 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:59:34.882980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:59:34.883044 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:59:34.883110 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:59:34.884257 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:59:34.885040 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:59:34.886420 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:59:34.886747 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:59:34.886831 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:59:34.890937 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:59:34.890942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:59:34.890945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:59:34.890986 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:59:34.892903 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:59:34.893554 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:59:34.893559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:59:34.893561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:59:34.893678 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:59:34.898464 (+ 4786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:59:34.898470 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:59:34.898472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:59:34.898823 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:59:34.898998 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 17:59:34.899480 (+ 482us) server_negotiation.cc:299] Negotiation successful
0929 17:59:34.899644 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 17:59:34.908131 18872 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:59:34.915920 18872 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:59:34.935165 19086 heartbeater.cc:503] Master 127.7.118.62:43963 was elected leader, sending a full tablet report...
I20260929 17:59:34.997355 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:59:34.979425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:39167 (local address 127.0.0.1:46106)
0929 17:59:34.981678 (+ 2253us) negotiation.cc:107] Waiting for socket to connect
0929 17:59:34.981750 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 17:59:34.982023 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:59:34.989262 (+ 7239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:59:34.989270 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:59:34.989311 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:59:34.989484 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:59:34.989494 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:59:34.992248 (+ 2754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:59:34.992252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:59:34.993725 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:59:34.993738 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:59:34.996466 (+ 2728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:59:34.996471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:59:34.996638 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:59:34.996694 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 17:59:34.996829 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 17:59:34.997024 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1985,"thread_start_us":142,"threads_started":1}
I20260929 17:59:34.998178 19372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:59:34.980404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46106 (local address 127.7.118.1:39167)
0929 17:59:34.988538 (+ 8134us) server_negotiation.cc:206] Beginning negotiation
0929 17:59:34.988547 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 17:59:34.988582 (+ 35us) server_negotiation.cc:407] Connection header received
0929 17:59:34.988682 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:59:34.988688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:59:34.988983 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:59:34.989227 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:59:34.989674 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:59:34.992090 (+ 2416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:59:34.993903 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:59:34.996079 (+ 2176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:59:34.996205 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:59:34.996389 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 17:59:34.997846 (+ 1457us) server_negotiation.cc:299] Negotiation successful
0929 17:59:34.998003 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7974,"thread_start_us":241,"threads_started":1}
I20260929 17:59:35.000046 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:59:34.979539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:42007 (local address 127.0.0.1:41708)
0929 17:59:34.990145 (+ 10606us) negotiation.cc:107] Waiting for socket to connect
0929 17:59:34.990178 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 17:59:34.990456 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:59:34.991271 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:59:34.991274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:59:34.991292 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:59:34.991401 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:59:34.991407 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:59:34.993282 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:59:34.993285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:59:34.998158 (+ 4873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:59:34.998166 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:59:34.999603 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:59:34.999606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:59:34.999718 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:59:34.999742 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 17:59:34.999814 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 17:59:34.999894 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10433,"thread_start_us":86,"threads_started":1}
I20260929 17:59:35.000516 19371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:59:34.979812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41708 (local address 127.7.118.2:42007)
0929 17:59:34.989841 (+ 10029us) server_negotiation.cc:206] Beginning negotiation
0929 17:59:34.989848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:59:34.990329 (+ 481us) server_negotiation.cc:407] Connection header received
0929 17:59:34.990663 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:59:34.990671 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:59:34.990946 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:59:34.991088 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:59:34.991926 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:59:34.993126 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:59:34.998329 (+ 5203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:59:34.999258 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:59:34.999369 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:59:34.999533 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 17:59:35.000257 (+ 724us) server_negotiation.cc:299] Negotiation successful
0929 17:59:35.000369 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9920,"thread_start_us":202,"threads_started":1}
I20260929 17:59:35.003782 19374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:59:34.981139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43702 (local address 127.7.118.3:36127)
0929 17:59:34.989617 (+ 8478us) server_negotiation.cc:206] Beginning negotiation
0929 17:59:34.989629 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:59:34.991738 (+ 2109us) server_negotiation.cc:407] Connection header received
0929 17:59:34.994440 (+ 2702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:59:34.994449 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:59:34.994724 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:59:34.994872 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:59:34.995316 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:59:34.996490 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:59:35.000971 (+ 4481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:59:35.001929 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:59:35.002027 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:59:35.002211 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 17:59:35.003370 (+ 1159us) server_negotiation.cc:299] Negotiation successful
0929 17:59:35.003625 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8348,"thread_start_us":106,"threads_started":1}
I20260929 17:59:35.003228 19370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:59:34.980996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:36127 (local address 127.0.0.1:43702)
0929 17:59:34.991635 (+ 10639us) negotiation.cc:107] Waiting for socket to connect
0929 17:59:34.991649 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:59:34.991811 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:59:34.995063 (+ 3252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:59:34.995067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:59:34.995088 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:59:34.995188 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:59:34.995193 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:59:34.996640 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:59:34.996643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:59:34.997868 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:59:34.997881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:59:35.002564 (+ 4683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:59:35.002571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:59:35.002689 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:59:35.002720 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 17:59:35.002822 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 17:59:35.002920 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10567,"thread_start_us":104,"threads_started":1}
I20260929 17:59:35.005036 19012 tablet_service.cc:1467] Processing CreateTablet for tablet f050447c2d0c444fb1709ee6f5ce07e5 (DEFAULT_TABLE table=test-table [id=12e5e16a38b647588ace68227e86e459]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:59:35.006754 19012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f050447c2d0c444fb1709ee6f5ce07e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:59:35.007798 19147 tablet_service.cc:1467] Processing CreateTablet for tablet f050447c2d0c444fb1709ee6f5ce07e5 (DEFAULT_TABLE table=test-table [id=12e5e16a38b647588ace68227e86e459]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:59:35.007805 19290 tablet_service.cc:1467] Processing CreateTablet for tablet f050447c2d0c444fb1709ee6f5ce07e5 (DEFAULT_TABLE table=test-table [id=12e5e16a38b647588ace68227e86e459]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:59:35.009047 19290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f050447c2d0c444fb1709ee6f5ce07e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:59:35.009047 19147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f050447c2d0c444fb1709ee6f5ce07e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:59:35.019294 19375 tablet_bootstrap.cc:492] T f050447c2d0c444fb1709ee6f5ce07e5 P 51134b442e0e4bb7abc492e42896d242: Bootstrap starting.
I20260929 17:59:35.019991 19376 tablet_bootstrap.cc:492] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6: Bootstrap starting.
I20260929 17:59:35.022554 19377 tablet_bootstrap.cc:492] T f050447c2d0c444fb1709ee6f5ce07e5 P d2bbe34e12a24b8c90c8822acc7517b1: Bootstrap starting.
Sep 29 17:59:35 dist-test-slave-r7ph krb5kdc[18396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704773, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for HTTP/127.7.118.20@KRBTEST.COM
I20260929 17:59:35.024155 19376 tablet_bootstrap.cc:654] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6: Neither blocks nor log segments found. Creating new log.
I20260929 17:59:35.025378 19376 log.cc:824] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6: Log is configured to *not* fsync() on all Append() calls
I20260929 17:59:35.025621 19375 tablet_bootstrap.cc:654] T f050447c2d0c444fb1709ee6f5ce07e5 P 51134b442e0e4bb7abc492e42896d242: Neither blocks nor log segments found. Creating new log.
I20260929 17:59:35.026767 19375 log.cc:824] T f050447c2d0c444fb1709ee6f5ce07e5 P 51134b442e0e4bb7abc492e42896d242: Log is configured to *not* fsync() on all Append() calls
I20260929 17:59:35.030620 19377 tablet_bootstrap.cc:654] T f050447c2d0c444fb1709ee6f5ce07e5 P d2bbe34e12a24b8c90c8822acc7517b1: Neither blocks nor log segments found. Creating new log.
I20260929 17:59:35.031769 19377 log.cc:824] T f050447c2d0c444fb1709ee6f5ce07e5 P d2bbe34e12a24b8c90c8822acc7517b1: Log is configured to *not* fsync() on all Append() calls
I20260929 17:59:35.032881 19375 tablet_bootstrap.cc:492] T f050447c2d0c444fb1709ee6f5ce07e5 P 51134b442e0e4bb7abc492e42896d242: No bootstrap required, opened a new log
I20260929 17:59:35.033345 19375 ts_tablet_manager.cc:1397] T f050447c2d0c444fb1709ee6f5ce07e5 P 51134b442e0e4bb7abc492e42896d242: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20260929 17:59:35.038904 19376 tablet_bootstrap.cc:492] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6: No bootstrap required, opened a new log
I20260929 17:59:35.038902 19375 raft_consensus.cc:348] T f050447c2d0c444fb1709ee6f5ce07e5 P 51134b442e0e4bb7abc492e42896d242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bbe34e12a24b8c90c8822acc7517b1" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 39167 } } peers { permanent_uuid: "4fcaa158682949c18a7841b9a36972b6" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42007 } } peers { permanent_uuid: "51134b442e0e4bb7abc492e42896d242" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36127 } }
I20260929 17:59:35.039652 19375 raft_consensus.cc:374] T f050447c2d0c444fb1709ee6f5ce07e5 P 51134b442e0e4bb7abc492e42896d242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:59:35.039790 19375 raft_consensus.cc:729] T f050447c2d0c444fb1709ee6f5ce07e5 P 51134b442e0e4bb7abc492e42896d242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51134b442e0e4bb7abc492e42896d242, State: Initialized, Role: FOLLOWER
I20260929 17:59:35.040530 19376 ts_tablet_manager.cc:1397] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.000s
I20260929 17:59:35.040499 19375 consensus_queue.cc:260] T f050447c2d0c444fb1709ee6f5ce07e5 P 51134b442e0e4bb7abc492e42896d242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bbe34e12a24b8c90c8822acc7517b1" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 39167 } } peers { permanent_uuid: "4fcaa158682949c18a7841b9a36972b6" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42007 } } peers { permanent_uuid: "51134b442e0e4bb7abc492e42896d242" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36127 } }
I20260929 17:59:35.042079 19375 ts_tablet_manager.cc:1428] T f050447c2d0c444fb1709ee6f5ce07e5 P 51134b442e0e4bb7abc492e42896d242: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260929 17:59:35.042634 19356 heartbeater.cc:503] Master 127.7.118.62:43963 was elected leader, sending a full tablet report...
W20260929 17:59:35.046979 19357 tablet.cc:2369] T f050447c2d0c444fb1709ee6f5ce07e5 P 51134b442e0e4bb7abc492e42896d242: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:59:35.049469 19376 raft_consensus.cc:348] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bbe34e12a24b8c90c8822acc7517b1" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 39167 } } peers { permanent_uuid: "4fcaa158682949c18a7841b9a36972b6" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42007 } } peers { permanent_uuid: "51134b442e0e4bb7abc492e42896d242" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36127 } }
I20260929 17:59:35.049988 19376 raft_consensus.cc:374] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:59:35.050102 19376 raft_consensus.cc:729] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fcaa158682949c18a7841b9a36972b6, State: Initialized, Role: FOLLOWER
I20260929 17:59:35.050865 19376 consensus_queue.cc:260] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bbe34e12a24b8c90c8822acc7517b1" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 39167 } } peers { permanent_uuid: "4fcaa158682949c18a7841b9a36972b6" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42007 } } peers { permanent_uuid: "51134b442e0e4bb7abc492e42896d242" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36127 } }
I20260929 17:59:35.052340 19376 ts_tablet_manager.cc:1428] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260929 17:59:35.052685 19377 tablet_bootstrap.cc:492] T f050447c2d0c444fb1709ee6f5ce07e5 P d2bbe34e12a24b8c90c8822acc7517b1: No bootstrap required, opened a new log
I20260929 17:59:35.053150 19377 ts_tablet_manager.cc:1397] T f050447c2d0c444fb1709ee6f5ce07e5 P d2bbe34e12a24b8c90c8822acc7517b1: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.002s
I20260929 17:59:35.053519 19222 heartbeater.cc:503] Master 127.7.118.62:43963 was elected leader, sending a full tablet report...
I20260929 17:59:35.058423 19377 raft_consensus.cc:348] T f050447c2d0c444fb1709ee6f5ce07e5 P d2bbe34e12a24b8c90c8822acc7517b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bbe34e12a24b8c90c8822acc7517b1" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 39167 } } peers { permanent_uuid: "4fcaa158682949c18a7841b9a36972b6" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42007 } } peers { permanent_uuid: "51134b442e0e4bb7abc492e42896d242" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36127 } }
I20260929 17:59:35.058820 19377 raft_consensus.cc:374] T f050447c2d0c444fb1709ee6f5ce07e5 P d2bbe34e12a24b8c90c8822acc7517b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:59:35.058890 19377 raft_consensus.cc:729] T f050447c2d0c444fb1709ee6f5ce07e5 P d2bbe34e12a24b8c90c8822acc7517b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2bbe34e12a24b8c90c8822acc7517b1, State: Initialized, Role: FOLLOWER
I20260929 17:59:35.059536 19377 consensus_queue.cc:260] T f050447c2d0c444fb1709ee6f5ce07e5 P d2bbe34e12a24b8c90c8822acc7517b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bbe34e12a24b8c90c8822acc7517b1" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 39167 } } peers { permanent_uuid: "4fcaa158682949c18a7841b9a36972b6" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42007 } } peers { permanent_uuid: "51134b442e0e4bb7abc492e42896d242" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36127 } }
I20260929 17:59:35.060818 19377 ts_tablet_manager.cc:1428] T f050447c2d0c444fb1709ee6f5ce07e5 P d2bbe34e12a24b8c90c8822acc7517b1: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20260929 17:59:35.116076 19382 raft_consensus.cc:482] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:59:35.116319 19382 raft_consensus.cc:504] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bbe34e12a24b8c90c8822acc7517b1" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 39167 } } peers { permanent_uuid: "4fcaa158682949c18a7841b9a36972b6" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42007 } } peers { permanent_uuid: "51134b442e0e4bb7abc492e42896d242" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36127 } }
I20260929 17:59:35.118021 19382 leader_election.cc:288] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2bbe34e12a24b8c90c8822acc7517b1 (127.7.118.1:39167), 51134b442e0e4bb7abc492e42896d242 (127.7.118.3:36127)
I20260929 17:59:35.134915 19372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:59:35.118748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:32863 (local address 127.7.118.1:39167)
0929 17:59:35.118875 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 17:59:35.118880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:59:35.119766 (+ 886us) server_negotiation.cc:407] Connection header received
0929 17:59:35.120291 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:59:35.120295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:59:35.120361 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:59:35.120427 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:59:35.125594 (+ 5167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:59:35.126359 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:59:35.133310 (+ 6951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:59:35.133872 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:59:35.133985 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:59:35.134074 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 17:59:35.134653 (+ 579us) server_negotiation.cc:299] Negotiation successful
0929 17:59:35.134768 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 17:59:35.135838 19385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:59:35.118290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:39167 (local address 127.7.118.2:32863)
0929 17:59:35.119659 (+ 1369us) negotiation.cc:107] Waiting for socket to connect
0929 17:59:35.119673 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:59:35.119852 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:59:35.122369 (+ 2517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:59:35.122373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:59:35.125029 (+ 2656us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:59:35.125386 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:59:35.125401 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:59:35.128986 (+ 3585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:59:35.128989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:59:35.130832 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:59:35.130849 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:59:35.134358 (+ 3509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:59:35.134362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:59:35.134478 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:59:35.134507 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 17:59:35.135521 (+ 1014us) client_negotiation.cc:240] Negotiation successful
0929 17:59:35.135644 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1297,"thread_start_us":131,"threads_started":1}
I20260929 17:59:35.137073 19040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f050447c2d0c444fb1709ee6f5ce07e5" candidate_uuid: "4fcaa158682949c18a7841b9a36972b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2bbe34e12a24b8c90c8822acc7517b1" is_pre_election: true
I20260929 17:59:35.137279 19374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:59:35.119342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:33187 (local address 127.7.118.3:36127)
0929 17:59:35.119457 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 17:59:35.119461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:59:35.126007 (+ 6546us) server_negotiation.cc:407] Connection header received
0929 17:59:35.126206 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:59:35.126209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:59:35.126284 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:59:35.126332 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:59:35.129679 (+ 3347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:59:35.130589 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:59:35.132355 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:59:35.133153 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:59:35.133263 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:59:35.133349 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 17:59:35.137036 (+ 3687us) server_negotiation.cc:299] Negotiation successful
0929 17:59:35.137156 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 17:59:35.137583 19040 raft_consensus.cc:2454] T f050447c2d0c444fb1709ee6f5ce07e5 P d2bbe34e12a24b8c90c8822acc7517b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fcaa158682949c18a7841b9a36972b6 in term 0.
I20260929 17:59:35.138221 19110 leader_election.cc:302] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fcaa158682949c18a7841b9a36972b6, d2bbe34e12a24b8c90c8822acc7517b1; no voters:
I20260929 17:59:35.138595 19382 raft_consensus.cc:2781] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:59:35.138708 19382 raft_consensus.cc:482] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:59:35.138769 19382 raft_consensus.cc:3037] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:59:35.139057 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:59:35.118659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:36127 (local address 127.7.118.2:33187)
0929 17:59:35.125905 (+ 7246us) negotiation.cc:107] Waiting for socket to connect
0929 17:59:35.125922 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:59:35.126089 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:59:35.126500 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:59:35.126503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:59:35.126523 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:59:35.126815 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:59:35.126824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:59:35.130923 (+ 4099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:59:35.130926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:59:35.132103 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:59:35.132119 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:59:35.136683 (+ 4564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:59:35.136687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:59:35.136815 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:59:35.136863 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 17:59:35.138777 (+ 1914us) client_negotiation.cc:240] Negotiation successful
0929 17:59:35.138890 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7139,"thread_start_us":145,"threads_started":1}
I20260929 17:59:35.139778 19310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f050447c2d0c444fb1709ee6f5ce07e5" candidate_uuid: "4fcaa158682949c18a7841b9a36972b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51134b442e0e4bb7abc492e42896d242" is_pre_election: true
I20260929 17:59:35.140121 19310 raft_consensus.cc:2454] T f050447c2d0c444fb1709ee6f5ce07e5 P 51134b442e0e4bb7abc492e42896d242 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fcaa158682949c18a7841b9a36972b6 in term 0.
I20260929 17:59:35.140210 19382 raft_consensus.cc:504] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bbe34e12a24b8c90c8822acc7517b1" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 39167 } } peers { permanent_uuid: "4fcaa158682949c18a7841b9a36972b6" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42007 } } peers { permanent_uuid: "51134b442e0e4bb7abc492e42896d242" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36127 } }
I20260929 17:59:35.140668 19382 leader_election.cc:288] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6 [CANDIDATE]: Term 1 election: Requested vote from peers d2bbe34e12a24b8c90c8822acc7517b1 (127.7.118.1:39167), 51134b442e0e4bb7abc492e42896d242 (127.7.118.3:36127)
I20260929 17:59:35.141145 19040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f050447c2d0c444fb1709ee6f5ce07e5" candidate_uuid: "4fcaa158682949c18a7841b9a36972b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2bbe34e12a24b8c90c8822acc7517b1"
I20260929 17:59:35.141242 19310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f050447c2d0c444fb1709ee6f5ce07e5" candidate_uuid: "4fcaa158682949c18a7841b9a36972b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51134b442e0e4bb7abc492e42896d242"
I20260929 17:59:35.141297 19040 raft_consensus.cc:3037] T f050447c2d0c444fb1709ee6f5ce07e5 P d2bbe34e12a24b8c90c8822acc7517b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:59:35.141376 19310 raft_consensus.cc:3037] T f050447c2d0c444fb1709ee6f5ce07e5 P 51134b442e0e4bb7abc492e42896d242 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:59:35.142880 19310 raft_consensus.cc:2454] T f050447c2d0c444fb1709ee6f5ce07e5 P 51134b442e0e4bb7abc492e42896d242 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fcaa158682949c18a7841b9a36972b6 in term 1.
I20260929 17:59:35.143191 19040 raft_consensus.cc:2454] T f050447c2d0c444fb1709ee6f5ce07e5 P d2bbe34e12a24b8c90c8822acc7517b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fcaa158682949c18a7841b9a36972b6 in term 1.
I20260929 17:59:35.143582 19109 leader_election.cc:302] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fcaa158682949c18a7841b9a36972b6, 51134b442e0e4bb7abc492e42896d242; no voters:
I20260929 17:59:35.144102 19382 raft_consensus.cc:2781] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:59:35.144342 19382 raft_consensus.cc:686] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6 [term 1 LEADER]: Becoming Leader. State: Replica: 4fcaa158682949c18a7841b9a36972b6, State: Running, Role: LEADER
I20260929 17:59:35.144740 19382 consensus_queue.cc:237] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bbe34e12a24b8c90c8822acc7517b1" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 39167 } } peers { permanent_uuid: "4fcaa158682949c18a7841b9a36972b6" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42007 } } peers { permanent_uuid: "51134b442e0e4bb7abc492e42896d242" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36127 } }
I20260929 17:59:35.148700 18872 catalog_manager.cc:5495] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fcaa158682949c18a7841b9a36972b6 (127.7.118.2). New cstate: current_term: 1 leader_uuid: "4fcaa158682949c18a7841b9a36972b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bbe34e12a24b8c90c8822acc7517b1" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 39167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fcaa158682949c18a7841b9a36972b6" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 42007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51134b442e0e4bb7abc492e42896d242" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36127 } health_report { overall_health: UNKNOWN } } }
W20260929 17:59:35.156440 19087 tablet.cc:2369] T f050447c2d0c444fb1709ee6f5ce07e5 P d2bbe34e12a24b8c90c8822acc7517b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:59:35.205971 19371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:59:35.200095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41724 (local address 127.7.118.2:42007)
0929 17:59:35.200311 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 17:59:35.200316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:59:35.200332 (+ 16us) server_negotiation.cc:407] Connection header received
0929 17:59:35.200429 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:59:35.200433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:59:35.200527 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:59:35.200652 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:59:35.201088 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:59:35.202003 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:59:35.203403 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:59:35.203770 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:59:35.203888 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:59:35.204386 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:59:35.205025 (+ 639us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:59:35.205161 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 17:59:35.205737 (+ 576us) server_negotiation.cc:299] Negotiation successful
0929 17:59:35.205809 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260929 17:59:35.214601 19310 raft_consensus.cc:1261] T f050447c2d0c444fb1709ee6f5ce07e5 P 51134b442e0e4bb7abc492e42896d242 [term 1 FOLLOWER]: Refusing update from remote peer 4fcaa158682949c18a7841b9a36972b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:59:35.215762 19382 consensus_queue.cc:1035] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51134b442e0e4bb7abc492e42896d242" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:59:35.225710 19040 raft_consensus.cc:1261] T f050447c2d0c444fb1709ee6f5ce07e5 P d2bbe34e12a24b8c90c8822acc7517b1 [term 1 FOLLOWER]: Refusing update from remote peer 4fcaa158682949c18a7841b9a36972b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:59:35.226728 19382 consensus_queue.cc:1035] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2bbe34e12a24b8c90c8822acc7517b1" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 39167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:59:35.240871 19391 mvcc.cc:205] Tried to move back new op lower bound from 7334726759268216832 to 7334726758996357120. Current Snapshot: MvccSnapshot[applied={T|T < 7334726759268216832}]
I20260929 17:59:35.248678 18872 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:59:35.248883 18872 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:59:35.252286 19393 mvcc.cc:205] Tried to move back new op lower bound from 7334726759268216832 to 7334726758996357120. Current Snapshot: MvccSnapshot[applied={T|T < 7334726759268216832}]
I20260929 17:59:35.252774 18872 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ccf9e85585d4673b8eaca6ed69d8331: Sending DeleteTablet for 3 replicas of tablet f050447c2d0c444fb1709ee6f5ce07e5
I20260929 17:59:35.254679 19012 tablet_service.cc:1514] Processing DeleteTablet for tablet f050447c2d0c444fb1709ee6f5ce07e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:59:35 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:46106
I20260929 17:59:35.255411 19147 tablet_service.cc:1514] Processing DeleteTablet for tablet f050447c2d0c444fb1709ee6f5ce07e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:59:35 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:41708
I20260929 17:59:35.256525 19290 tablet_service.cc:1514] Processing DeleteTablet for tablet f050447c2d0c444fb1709ee6f5ce07e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:59:35 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:43702
I20260929 17:59:35.257927 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 18953
I20260929 17:59:35.266674 19400 tablet_replica.cc:330] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6: stopping tablet replica
I20260929 17:59:35.267220 19400 raft_consensus.cc:2229] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:59:35.267828 19400 raft_consensus.cc:2258] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:59:35.268167 19401 tablet_replica.cc:330] T f050447c2d0c444fb1709ee6f5ce07e5 P 51134b442e0e4bb7abc492e42896d242: stopping tablet replica
I20260929 17:59:35.268800 19401 raft_consensus.cc:2229] T f050447c2d0c444fb1709ee6f5ce07e5 P 51134b442e0e4bb7abc492e42896d242 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:59:35.270556 19400 ts_tablet_manager.cc:1905] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:59:35.271061 19401 raft_consensus.cc:2258] T f050447c2d0c444fb1709ee6f5ce07e5 P 51134b442e0e4bb7abc492e42896d242 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:59:35.273072 19401 ts_tablet_manager.cc:1905] T f050447c2d0c444fb1709ee6f5ce07e5 P 51134b442e0e4bb7abc492e42896d242: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:59:35.276948 19400 ts_tablet_manager.cc:1918] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:59:35.280210 19401 ts_tablet_manager.cc:1918] T f050447c2d0c444fb1709ee6f5ce07e5 P 51134b442e0e4bb7abc492e42896d242: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260929 17:59:35.283005 18857 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.118.1:39167 (error 108)
I20260929 17:59:35.283318 19400 log.cc:1196] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-1/wal/wals/f050447c2d0c444fb1709ee6f5ce07e5
I20260929 17:59:35.283864 19400 ts_tablet_manager.cc:1939] T f050447c2d0c444fb1709ee6f5ce07e5 P 4fcaa158682949c18a7841b9a36972b6: Deleting consensus metadata
I20260929 17:59:35.284072 19401 log.cc:1196] T f050447c2d0c444fb1709ee6f5ce07e5 P 51134b442e0e4bb7abc492e42896d242: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704657566865-7640-0/minicluster-data/ts-2/wal/wals/f050447c2d0c444fb1709ee6f5ce07e5
I20260929 17:59:35.284513 19401 ts_tablet_manager.cc:1939] T f050447c2d0c444fb1709ee6f5ce07e5 P 51134b442e0e4bb7abc492e42896d242: Deleting consensus metadata
I20260929 17:59:35.286073 18856 catalog_manager.cc:4841] TS 4fcaa158682949c18a7841b9a36972b6 (127.7.118.2:42007): tablet f050447c2d0c444fb1709ee6f5ce07e5 (table test-table [id=12e5e16a38b647588ace68227e86e459]) successfully deleted
I20260929 17:59:35.290501 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 19092
I20260929 17:59:35.292341 19370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:59:35.291948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:39167 (local address 127.0.0.1:46110)
0929 17:59:35.292160 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 17:59:35.292242 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.1:39167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260929 17:59:35.292702 18857 catalog_manager.cc:4584] TS d2bbe34e12a24b8c90c8822acc7517b1 (127.7.118.1:39167): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f050447c2d0c444fb1709ee6f5ce07e5: Network error: Client connection negotiation failed: client connection to 127.7.118.1:39167: connect: Connection refused (error 111)
I20260929 17:59:35.293109 18856 catalog_manager.cc:4841] TS 51134b442e0e4bb7abc492e42896d242 (127.7.118.3:36127): tablet f050447c2d0c444fb1709ee6f5ce07e5 (table test-table [id=12e5e16a38b647588ace68227e86e459]) successfully deleted
I20260929 17:59:35.299897 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 19227
I20260929 17:59:35.307435 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 18840
I20260929 17:59:35.316458 7640 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 17:59:35.662736 7640 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 17:59:35.662942 7640 mini_ranger.cc:66] Stopping Ranger...
I20260929 17:59:36.009227 7640 mini_ranger.cc:68] Stopped Ranger
2026-09-29 17:59:36.009 UTC [18426] LOG: received smart shutdown request
2026-09-29 17:59:36.013 UTC [18426] LOG: background worker "logical replication launcher" (PID 18435) exited with exit code 1
2026-09-29 17:59:36.014 UTC [18430] LOG: shutting down
2026-09-29 17:59:36.252 UTC [18426] LOG: database system is shut down
2026-09-29T17:59:36Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52096 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:59:36 dist-test-slave-r7ph krb5kdc[19411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:59:36 dist-test-slave-r7ph krb5kdc[19411](info): set up 2 sockets
Sep 29 17:59:36 dist-test-slave-r7ph krb5kdc[19411](info): commencing operation
krb5kdc: starting...
W20260929 17:59:37.470546 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 17:59:37 dist-test-slave-r7ph krb5kdc[19411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:59:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:59:37Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.7.118.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.7.118.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.7.118.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/rangeradmin_127.7.118.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.7.118.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/rangeradmin_127.7.118.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.7.118.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.7.118.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.7.118.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/rangerlookup_127.7.118.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.7.118.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/rangerlookup_127.7.118.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.118.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.7.118.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.118.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/HTTP_127.7.118.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.118.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/HTTP_127.7.118.20@KRBTEST.COM.keytab.
I20260929 17:59:37.678501 7640 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task8_cVK5/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/postgres -l logfile start
2026-09-29 17:59:40.810 UTC [19441] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 17:59:40.810 UTC [19441] LOG: listening on IPv4 address "127.7.118.20", port 57857
2026-09-29 17:59:40.815 UTC [19441] LOG: listening on Unix socket "/tmp/.s.PGSQL.57857"
2026-09-29 17:59:40.838 UTC [19444] LOG: database system was shut down at 2026-09-29 17:59:38 UTC
2026-09-29 17:59:40.843 UTC [19441] LOG: database system is ready to accept connections
I20260929 17:59:41.794293 7640 mini_postgres.cc:96] Postgres bound to 57857
2026-09-29 17:59:41.800 UTC [19452] FATAL: database "slave" does not exist
127.7.118.20:57857 - accepting connections
I20260929 17:59:41.801551 7640 mini_ranger.cc:161] Starting Ranger...
I20260929 17:59:41.817899 7640 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 17:59:42.631099 7640 mini_ranger.cc:87] Created ranger Postgres database
I20260929 17:59:42.631212 7640 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ranger-admin
2026-09-29 17:59:42,916 [I] DB FLAVOR :POSTGRES
2026-09-29 17:59:42,917 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 17:59:42,917 [I] Checking connection..
2026-09-29 17:59:42,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:57857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 17:59:43,201 [I] Checking connection passed.
2026-09-29 17:59:43,202 [I] --------- Verifying version history table ---------
2026-09-29 17:59:43,202 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:57857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 17:59:43,475 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 17:59:43,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:57857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 17:59:43,746 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 17:59:43,746 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 17:59:43,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:57857/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 17:59:43.990 UTC [19516] WARNING: there is no transaction in progress
2026-09-29 17:59:43,997 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 17:59:43,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:57857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 17:59:44,282 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 17:59:44,283 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 17:59:44,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:57857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 17:59:44,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:57857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 17:59:44,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:57857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-r7ph.c.gcp-upstream.internal','N') ;"
2026-09-29 17:59:45,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:57857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 17:59:45,361 [I] Table x_portal_user does not exist in database ranger
2026-09-29 17:59:45,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:57857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 17:59:45,642 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 17:59:45,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:57857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 17:59:45,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:57857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 17:59:46,195 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 17:59:46,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:57857/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 17:59:47.535 UTC [19624] WARNING: there is no transaction in progress
2026-09-29 17:59:47.559 UTC [19624] WARNING: there is no transaction in progress
2026-09-29 17:59:47.585 UTC [19624] WARNING: there is no transaction in progress
2026-09-29 17:59:47.613 UTC [19624] WARNING: there is no transaction in progress
2026-09-29 17:59:47.645 UTC [19624] WARNING: there is no transaction in progress
2026-09-29 17:59:47.672 UTC [19624] WARNING: there is no transaction in progress
2026-09-29 17:59:47.743 UTC [19624] WARNING: there is no transaction in progress
2026-09-29 17:59:47.753 UTC [19624] WARNING: there is no transaction in progress
2026-09-29 17:59:47.776 UTC [19624] WARNING: there is no transaction in progress
2026-09-29 17:59:47.793 UTC [19624] WARNING: there is no transaction in progress
2026-09-29 17:59:47.810 UTC [19624] WARNING: there is no transaction in progress
2026-09-29 17:59:47.832 UTC [19624] WARNING: there is no transaction in progress
2026-09-29 17:59:47.847 UTC [19624] WARNING: there is no transaction in progress
2026-09-29 17:59:47.863 UTC [19624] WARNING: there is no transaction in progress
2026-09-29 17:59:47.876 UTC [19624] WARNING: there is no transaction in progress
2026-09-29 17:59:47.888 UTC [19624] WARNING: there is no transaction in progress
2026-09-29 17:59:48,994 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 17:59:48,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:57857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 17:59:49,251 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 17:59:49,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:57857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 17:59:49,554 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 17:59:49,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:57857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 17:59:49,829 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 17:59:49,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:57857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 17:59:50,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:57857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 17:59:50,388 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:57857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-r7ph.c.gcp-upstream.internal';"
2026-09-29 17:59:50,673 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 17:59:50,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:57857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 17:59:50,948 [I] DB_PATCHES have already been applied
I20260929 17:59:50.954447 7640 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ranger-admin:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task8_cVK5/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 17:59:50.954523 7640 mini_ranger.cc:193] Using host: 127.7.118.20
I20260929 17:59:50.958201 7640 mini_ranger.cc:227] Ranger admin URL: http://127.7.118.20:47193
Sep 29, 2026 5:59:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 5:59:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 5:59:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 5:59:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 5:59:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 5:59:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 5:59:51 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 17:59:52.132974 7640 mini_ranger.cc:160] Time spent starting Ranger: real 10.331s user 0.000s sys 0.011s
Sep 29, 2026 6:00:15 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 6:00:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 6:00:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 6:00:15 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 6:00:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 6:00:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 6:00:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 6:00:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 6:00:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 6:00:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 6:00:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 6:00:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 6:00:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 6:00:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 6:00:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 6:00:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 6:00:15 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 6:00:16 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 18:00:17.140085 7640 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.7.118.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.7.118.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.7.118.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/rangerkms_127.7.118.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.7.118.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/rangerkms_127.7.118.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.118.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.7.118.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.118.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/HTTP_127.7.118.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.118.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/HTTP_127.7.118.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 18:00:17 dist-test-slave-r7ph krb5kdc[19411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704817, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 18:00:17.237687 7640 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 18:00:17.250101 7640 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 18:00:18.266626 7640 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 18:00:18.266703 7640 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ranger-kms
I20260929 18:00:18.266717 7640 mini_ranger_kms.cc:227] Using postgres at 127.7.118.20:57857
2026-09-29 18:00:18,445 [I] DB FLAVOR :POSTGRES
2026-09-29 18:00:18,445 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 18:00:18,445 [I] Checking connection
2026-09-29 18:00:18,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:57857/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 18:00:18,703 [I] connection success
2026-09-29 18:00:18,703 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 18:00:18,703 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 18:00:18,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:57857/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 18:00:18,995 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 18:00:18,995 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 18:00:18,995 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 18:00:18,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.118.20:57857/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 18:00:19,320 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 18:00:19.885334 7640 mini_ranger_kms.cc:310] Created kms service
I20260929 18:00:20.104867 7640 mini_ranger_kms.cc:326] Created kudu user
I20260929 18:00:20.158306 7640 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 18:00:20 dist-test-slave-r7ph krb5kdc[19411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704817, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.7.118.20@KRBTEST.COM
I20260929 18:00:20.654731 7640 mini_ranger_kms.cc:384] Added ranger policy
I20260929 18:00:20.654908 7640 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ranger-kms:/tmp/dist-test-task8_cVK5/build/debug/bin/postgresql.jar:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task8_cVK5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 18:00:20.654932 7640 mini_ranger_kms.cc:240] Using host: 127.7.118.20
I20260929 18:00:20.659703 7640 mini_ranger_kms.cc:276] Ranger KMS PID: 19805
I20260929 18:00:20.659847 7640 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.7.118.20:59261
Sep 29, 2026 6:00:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 6:00:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 6:00:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 6:00:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 6:00:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 6:00:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 29 18:00:22 dist-test-slave-r7ph krb5kdc[19411](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790704822, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.118.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:23.035318 7640 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.798s user 0.000s sys 0.164s
I20260929 18:00:23.035449 7640 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 18:00:23.035517 7640 mini_ranger_kms.cc:401] 127.7.118.20:59261/kms/v1/keys
Sep 29 18:00:23 dist-test-slave-r7ph krb5kdc[19411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704822, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.118.20@KRBTEST.COM for HTTP/127.7.118.20@KRBTEST.COM
Sep 29 18:00:23 dist-test-slave-r7ph krb5kdc[19411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704822, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.118.20@KRBTEST.COM for HTTP/127.7.118.20@KRBTEST.COM
Sep 29 18:00:26 dist-test-slave-r7ph krb5kdc[19411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 18:00:26.182900 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:41935
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:41935
--ranger_config_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.118.20:59261/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:00:26.301403 19855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:26.301707 19855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:26.301748 19855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:00:26.306035 19855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:00:26.306123 19855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:26.306143 19855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:26.306159 19855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:00:26.306174 19855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:00:26.312224 19855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/wal
--ranger_kms_url=127.7.118.20:59261/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:41935
--ranger_config_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:41935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:26.313413 19855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:26.314553 19855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:00:26.324774 19860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:26.324762 19861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:26.324759 19863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:26.324820 19855 server_base.cc:1034] running on GCE node
I20260929 18:00:26.325815 19855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:26.327234 19855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:26.328495 19855 hybrid_clock.cc:648] HybridClock initialized: now 1790704826328456 us; error 58 us; skew 500 ppm
Sep 29 18:00:26 dist-test-slave-r7ph krb5kdc[19411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704826, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:26.336697 19855 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 18:00:26.337917 19855 webserver.cc:460] Webserver started at http://127.7.118.62:42543/ using document root <none> and password file <none>
I20260929 18:00:26.338642 19855 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:26.338713 19855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:00:26.338959 19855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:00:26 dist-test-slave-r7ph krb5kdc[19411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704826, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for HTTP/127.7.118.20@KRBTEST.COM
I20260929 18:00:26.399686 19855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "c492e5d3289b413cb11b1e1a5b64347f"
format_stamp: "Formatted at 2026-09-29 18:00:26 on dist-test-slave-r7ph"
server_key: "d7420b61e2d9b5de3644b0c78ef2e71c"
server_key_iv: "beead2d3454543583aff7e4ff76e2b39"
server_key_version: "kuduclusterkey@0"
I20260929 18:00:26.400520 19855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "c492e5d3289b413cb11b1e1a5b64347f"
format_stamp: "Formatted at 2026-09-29 18:00:26 on dist-test-slave-r7ph"
server_key: "d7420b61e2d9b5de3644b0c78ef2e71c"
server_key_iv: "beead2d3454543583aff7e4ff76e2b39"
server_key_version: "kuduclusterkey@0"
I20260929 18:00:26.406140 19855 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 18:00:26.425261 19870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:26.427259 19855 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 18:00:26.427462 19855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "c492e5d3289b413cb11b1e1a5b64347f"
format_stamp: "Formatted at 2026-09-29 18:00:26 on dist-test-slave-r7ph"
server_key: "d7420b61e2d9b5de3644b0c78ef2e71c"
server_key_iv: "beead2d3454543583aff7e4ff76e2b39"
server_key_version: "kuduclusterkey@0"
I20260929 18:00:26.427625 19855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:26.461349 19855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:26.465039 19855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:26.465303 19855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:26.473817 19855 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:41935
I20260929 18:00:26.473817 19922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:41935 every 8 connection(s)
I20260929 18:00:26.475145 19855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:00:26.478322 19923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:00:26.482511 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 19855
I20260929 18:00:26.482667 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:00:26.482997 7640 external_mini_cluster.cc:1428] Setting key fd68214bc8f39ff41c6e9aeda4d8cd36
I20260929 18:00:26.484992 19923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f: Bootstrap starting.
I20260929 18:00:26.487793 19923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f: Neither blocks nor log segments found. Creating new log.
I20260929 18:00:26.488700 19923 log.cc:824] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:00:26.491146 19923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f: No bootstrap required, opened a new log
Sep 29 18:00:26 dist-test-slave-r7ph krb5kdc[19411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:00:26.496690 19923 raft_consensus.cc:348] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c492e5d3289b413cb11b1e1a5b64347f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 41935 } }
I20260929 18:00:26.497040 19923 raft_consensus.cc:374] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:00:26.497093 19923 raft_consensus.cc:729] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c492e5d3289b413cb11b1e1a5b64347f, State: Initialized, Role: FOLLOWER
I20260929 18:00:26.497704 19923 consensus_queue.cc:260] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c492e5d3289b413cb11b1e1a5b64347f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 41935 } }
I20260929 18:00:26.497884 19923 raft_consensus.cc:388] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:00:26.497977 19923 raft_consensus.cc:482] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:00:26.498085 19923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:00:26.499179 19923 raft_consensus.cc:504] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c492e5d3289b413cb11b1e1a5b64347f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 41935 } }
I20260929 18:00:26.499588 19923 leader_election.cc:302] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c492e5d3289b413cb11b1e1a5b64347f; no voters:
I20260929 18:00:26.499985 19923 leader_election.cc:288] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:00:26.500070 19928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:00:26.500455 19928 raft_consensus.cc:686] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [term 1 LEADER]: Becoming Leader. State: Replica: c492e5d3289b413cb11b1e1a5b64347f, State: Running, Role: LEADER
I20260929 18:00:26.500859 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:26.484608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47546 (local address 127.7.118.62:41935)
0929 18:00:26.485076 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:26.485098 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:26.485164 (+ 66us) server_negotiation.cc:407] Connection header received
0929 18:00:26.485860 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:26.485878 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:26.486311 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:26.486659 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:26.487638 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:26.488939 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:26.490189 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:26.490855 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:26.490997 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:26.494527 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:26.494536 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:26.494558 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:26.494603 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:26.496753 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:26.497523 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:26.497529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:26.497536 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:26.497682 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:26.498135 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:26.498139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:26.498140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:26.498688 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:26.498937 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:26.499248 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 18:00:26.499663 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":160,"threads_started":1}
I20260929 18:00:26.500950 19928 consensus_queue.cc:237] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c492e5d3289b413cb11b1e1a5b64347f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 41935 } }
I20260929 18:00:26.501744 19923 sys_catalog.cc:564] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:00:26.502938 19929 sys_catalog.cc:455] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c492e5d3289b413cb11b1e1a5b64347f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c492e5d3289b413cb11b1e1a5b64347f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 41935 } } }
I20260929 18:00:26.503201 19929 sys_catalog.cc:458] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [sys.catalog]: This master's current role is: LEADER
I20260929 18:00:26.503615 19930 sys_catalog.cc:455] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c492e5d3289b413cb11b1e1a5b64347f. Latest consensus state: current_term: 1 leader_uuid: "c492e5d3289b413cb11b1e1a5b64347f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c492e5d3289b413cb11b1e1a5b64347f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 41935 } } }
I20260929 18:00:26.503806 19930 sys_catalog.cc:458] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [sys.catalog]: This master's current role is: LEADER
I20260929 18:00:26.503921 19937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:00:26.507104 19937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:00:26.507645 19923 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 18:00:26.518181 19937 catalog_manager.cc:1347] Generated new cluster ID: 2b8af73882e04446be95d146a9afc4d7
I20260929 18:00:26.518409 19937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:00:26.532545 19937 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:00:26.533583 19937 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:00:26.556310 19937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f: Generated new TSK 0
I20260929 18:00:26.557236 19937 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.118.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
I20260929 18:00:27.320508 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.118.20:59261/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:0
--local_ip_for_outbound_sockets=127.7.118.1
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:41935
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:00:27.491330 19968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:27.491636 19968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:27.491686 19968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:00:27.499607 19968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:27.499724 19968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:27.499845 19968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 18:00:27.508674 19968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.7.118.20:59261/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:41935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:27.510000 19968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:27.511667 19968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:00:27.526000 19975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:27.526093 19976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:27.526000 19978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:27.526826 19968 server_base.cc:1034] running on GCE node
I20260929 18:00:27.527350 19968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:27.528172 19968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:27.529423 19968 hybrid_clock.cc:648] HybridClock initialized: now 1790704827529395 us; error 52 us; skew 500 ppm
Sep 29 18:00:27 dist-test-slave-r7ph krb5kdc[19411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704827, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:27.538424 19968 init.cc:377] Logged in from keytab as kudu/127.7.118.1@KRBTEST.COM (short username kudu)
I20260929 18:00:27.539796 19968 webserver.cc:460] Webserver started at http://127.7.118.1:34459/ using document root <none> and password file <none>
I20260929 18:00:27.540467 19968 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:27.540540 19968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:00:27.540786 19968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:00:27 dist-test-slave-r7ph krb5kdc[19411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704827, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for HTTP/127.7.118.20@KRBTEST.COM
I20260929 18:00:27.559244 19968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/data/instance:
uuid: "543cbcfe2b464c4da5209dd97a87c626"
format_stamp: "Formatted at 2026-09-29 18:00:27 on dist-test-slave-r7ph"
server_key: "020e8867a4b2aabcedbb78d5b9c86894"
server_key_iv: "5463d8f16b6d4c671434c96ffcaca951"
server_key_version: "kuduclusterkey@0"
I20260929 18:00:27.561638 19968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance:
uuid: "543cbcfe2b464c4da5209dd97a87c626"
format_stamp: "Formatted at 2026-09-29 18:00:27 on dist-test-slave-r7ph"
server_key: "020e8867a4b2aabcedbb78d5b9c86894"
server_key_iv: "5463d8f16b6d4c671434c96ffcaca951"
server_key_version: "kuduclusterkey@0"
I20260929 18:00:27.568430 19968 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 18:00:27.598802 19985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:27.600937 19968 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.003s sys 0.000s
I20260929 18:00:27.601220 19968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "543cbcfe2b464c4da5209dd97a87c626"
format_stamp: "Formatted at 2026-09-29 18:00:27 on dist-test-slave-r7ph"
server_key: "020e8867a4b2aabcedbb78d5b9c86894"
server_key_iv: "5463d8f16b6d4c671434c96ffcaca951"
server_key_version: "kuduclusterkey@0"
I20260929 18:00:27.601473 19968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:27.635784 19968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:27.640460 19968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:27.640766 19968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:27.641738 19968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:00:27.643131 19968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:00:27.643276 19968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 18:00:27.643422 19968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:00:27.643517 19968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:27.663676 19968 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:37903
I20260929 18:00:27.665421 19968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 18:00:27.666396 20100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:37903 every 8 connection(s)
I20260929 18:00:27.671975 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 19968
I20260929 18:00:27.672127 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance
I20260929 18:00:27.672550 7640 external_mini_cluster.cc:1428] Setting key 2824a24d8e988096c79152ff93e242be
WARNING: no policy specified for kudu/127.7.118.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.2@KRBTEST.COM" created.
Sep 29 18:00:27 dist-test-slave-r7ph krb5kdc[19411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704827, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:00:27.715981 20103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:27.669348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:41935 (local address 127.7.118.1:50389)
0929 18:00:27.670462 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:27.670540 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:27.671780 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:27.681467 (+ 9687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:27.681488 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:27.682038 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:00:27.683029 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:27.683059 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:27.684615 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:27.684620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:27.686577 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:27.686598 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:27.687554 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:27.687560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:27.687800 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:27.691445 (+ 3645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:00:27.691477 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:00:27.701490 (+ 10013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:00:27.711589 (+ 10099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:27.711598 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:27.711622 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:27.712133 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:27.712554 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:27.712560 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:27.712562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:27.712809 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:27.713414 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:00:27.713423 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:00:27.713922 (+ 499us) client_negotiation.cc:769] Sending connection context
0929 18:00:27.714170 (+ 248us) client_negotiation.cc:240] Negotiation successful
0929 18:00:27.714701 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":258,"threads_started":1}
I20260929 18:00:27.719884 20104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:27.670809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:50389 (local address 127.7.118.62:41935)
0929 18:00:27.680829 (+ 10020us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:27.680834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:27.680853 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:00:27.680898 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:27.680901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:27.680950 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:27.681069 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:27.683331 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:27.684355 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:27.686793 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:27.687185 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:27.687312 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:27.701787 (+ 14475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:27.701793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:27.701796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:27.701835 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:27.711336 (+ 9501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:27.712297 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:27.712303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:27.712307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:27.712429 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:27.712952 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:27.712955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:27.712958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:27.713236 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:27.713348 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:27.718838 (+ 5490us) server_negotiation.cc:299] Negotiation successful
0929 18:00:27.719608 (+ 770us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9927,"thread_start_us":97,"threads_started":1}
I20260929 18:00:27.724102 20101 heartbeater.cc:347] Connected to a master server at 127.7.118.62:41935
I20260929 18:00:27.724545 20101 heartbeater.cc:464] Registering TS with master...
I20260929 18:00:27.725292 20101 heartbeater.cc:511] Master 127.7.118.62:41935 requested a full tablet report, sending...
I20260929 18:00:27.727519 19887 ts_manager.cc:194] Registered new tserver with Master: 543cbcfe2b464c4da5209dd97a87c626 (127.7.118.1:37903)
I20260929 18:00:27.729336 19887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.1@KRBTEST.COM'} at 127.7.118.1:50389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
I20260929 18:00:27.735001 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.118.20:59261/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:0
--local_ip_for_outbound_sockets=127.7.118.2
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:41935
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:00:27 dist-test-slave-r7ph krb5kdc[19411](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790704827, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 18:00:27.902616 20107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:27.903050 20107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:27.903118 20107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:00:27.909729 20107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:27.909842 20107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:27.909938 20107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 18:00:27.918650 20107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.7.118.20:59261/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:41935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:27.920158 20107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:27.921557 20107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:00:27.935132 20116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:27.935452 20113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:27.935187 20114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:27.936733 20107 server_base.cc:1034] running on GCE node
I20260929 18:00:27.937292 20107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:27.938189 20107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:27.939518 20107 hybrid_clock.cc:648] HybridClock initialized: now 1790704827939479 us; error 58 us; skew 500 ppm
Sep 29 18:00:27 dist-test-slave-r7ph krb5kdc[19411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704827, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:27.943523 20107 init.cc:377] Logged in from keytab as kudu/127.7.118.2@KRBTEST.COM (short username kudu)
I20260929 18:00:27.944875 20107 webserver.cc:460] Webserver started at http://127.7.118.2:34819/ using document root <none> and password file <none>
I20260929 18:00:27.945639 20107 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:27.945744 20107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:00:27.946024 20107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:00:27 dist-test-slave-r7ph krb5kdc[19411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704827, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for HTTP/127.7.118.20@KRBTEST.COM
I20260929 18:00:27.962751 20107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/data/instance:
uuid: "7d3e234f366049d586a0803cc32d2d30"
format_stamp: "Formatted at 2026-09-29 18:00:27 on dist-test-slave-r7ph"
server_key: "f9e9bd4ac9c278bb87059798d7db2bb4"
server_key_iv: "916ffc915fa997c4fe7c53a977349fb6"
server_key_version: "kuduclusterkey@0"
I20260929 18:00:27.963680 20107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance:
uuid: "7d3e234f366049d586a0803cc32d2d30"
format_stamp: "Formatted at 2026-09-29 18:00:27 on dist-test-slave-r7ph"
server_key: "f9e9bd4ac9c278bb87059798d7db2bb4"
server_key_iv: "916ffc915fa997c4fe7c53a977349fb6"
server_key_version: "kuduclusterkey@0"
I20260929 18:00:27.969937 20107 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 18:00:27.990769 20123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:27.992982 20107 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260929 18:00:27.993181 20107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "7d3e234f366049d586a0803cc32d2d30"
format_stamp: "Formatted at 2026-09-29 18:00:27 on dist-test-slave-r7ph"
server_key: "f9e9bd4ac9c278bb87059798d7db2bb4"
server_key_iv: "916ffc915fa997c4fe7c53a977349fb6"
server_key_version: "kuduclusterkey@0"
I20260929 18:00:27.993399 20107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:28.015153 20107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:28.019196 20107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:28.019419 20107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:28.020174 20107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:00:28.021301 20107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:00:28.021381 20107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:28.021437 20107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:00:28.021459 20107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:28.032780 20107 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:40527
I20260929 18:00:28.033049 20236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:40527 every 8 connection(s)
I20260929 18:00:28.034045 20107 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 18:00:28.038079 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 20107
I20260929 18:00:28.038349 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance
I20260929 18:00:28.038704 7640 external_mini_cluster.cc:1428] Setting key d3c39760e3e85291ad2fbdb2fdf1019e
Sep 29 18:00:28 dist-test-slave-r7ph krb5kdc[19411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704827, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.118.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.3@KRBTEST.COM" created.
I20260929 18:00:28.060339 20239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:28.036853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:41935 (local address 127.7.118.2:44865)
0929 18:00:28.037482 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:28.037556 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:28.039011 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:28.039869 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:28.039884 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:28.040373 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:00:28.041456 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:28.041488 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:28.042866 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:28.042869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:28.044204 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:28.044223 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:28.045563 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:28.045569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:28.045965 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:28.046819 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:00:28.046839 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:00:28.048285 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:00:28.056063 (+ 7778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:28.056072 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:28.056096 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:28.056554 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:28.057134 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:28.057137 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:28.057140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:28.057410 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:28.058052 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:00:28.058061 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:00:28.058539 (+ 478us) client_negotiation.cc:769] Sending connection context
0929 18:00:28.058774 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 18:00:28.059177 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":165,"threads_started":1}
I20260929 18:00:28.060832 20104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:28.036295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:44865 (local address 127.7.118.62:41935)
0929 18:00:28.036456 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:28.036460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:28.037725 (+ 1265us) server_negotiation.cc:407] Connection header received
0929 18:00:28.039374 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:28.039379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:28.039448 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:28.039545 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:28.041697 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:28.042690 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:28.044397 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:28.044959 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:28.045442 (+ 483us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:28.048520 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:28.048524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:28.048526 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:28.048557 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:28.055821 (+ 7264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:28.056697 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:28.056701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:28.056703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:28.056811 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:28.057549 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:28.057553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:28.057555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:28.057827 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:28.057933 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:28.060177 (+ 2244us) server_negotiation.cc:299] Negotiation successful
0929 18:00:28.060362 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 18:00:28.062660 20237 heartbeater.cc:347] Connected to a master server at 127.7.118.62:41935
I20260929 18:00:28.062971 20237 heartbeater.cc:464] Registering TS with master...
I20260929 18:00:28.063687 20237 heartbeater.cc:511] Master 127.7.118.62:41935 requested a full tablet report, sending...
I20260929 18:00:28.065383 19887 ts_manager.cc:194] Registered new tserver with Master: 7d3e234f366049d586a0803cc32d2d30 (127.7.118.2:40527)
I20260929 18:00:28.066402 19887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.2@KRBTEST.COM'} at 127.7.118.2:44865
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
I20260929 18:00:28.086439 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.118.20:59261/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:0
--local_ip_for_outbound_sockets=127.7.118.3
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:41935
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:00:28 dist-test-slave-r7ph krb5kdc[19411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704827, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for HTTP/127.7.118.20@KRBTEST.COM
W20260929 18:00:28.236630 20242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:28.237121 20242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:28.237241 20242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:00:28.243700 20242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:28.243896 20242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:28.244036 20242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 18:00:28.252437 20242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.7.118.20:59261/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:41935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:28.254171 20242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:28.255807 20242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:00:28.270510 20250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:28.270977 20248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:28.271775 20242 server_base.cc:1034] running on GCE node
W20260929 18:00:28.274499 20247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:28.275071 20242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:28.275888 20242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:28.277093 20242 hybrid_clock.cc:648] HybridClock initialized: now 1790704828277053 us; error 60 us; skew 500 ppm
Sep 29 18:00:28 dist-test-slave-r7ph krb5kdc[19411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704828, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:28.280735 20242 init.cc:377] Logged in from keytab as kudu/127.7.118.3@KRBTEST.COM (short username kudu)
I20260929 18:00:28.281715 20242 webserver.cc:460] Webserver started at http://127.7.118.3:36057/ using document root <none> and password file <none>
I20260929 18:00:28.282449 20242 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:28.282550 20242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:00:28.282828 20242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:00:28 dist-test-slave-r7ph krb5kdc[19411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704828, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for HTTP/127.7.118.20@KRBTEST.COM
I20260929 18:00:28.297360 20242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/data/instance:
uuid: "b7d1a5a21de1419fb55434bf9207c1db"
format_stamp: "Formatted at 2026-09-29 18:00:28 on dist-test-slave-r7ph"
server_key: "3e9e69cda5a05d11168599ec1d8bbcf0"
server_key_iv: "b5f7ccc522c77c953d0c83dc12e2f144"
server_key_version: "kuduclusterkey@0"
I20260929 18:00:28.298350 20242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance:
uuid: "b7d1a5a21de1419fb55434bf9207c1db"
format_stamp: "Formatted at 2026-09-29 18:00:28 on dist-test-slave-r7ph"
server_key: "3e9e69cda5a05d11168599ec1d8bbcf0"
server_key_iv: "b5f7ccc522c77c953d0c83dc12e2f144"
server_key_version: "kuduclusterkey@0"
I20260929 18:00:28.308522 20242 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.005s sys 0.000s
I20260929 18:00:28.339526 20257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:28.341665 20242 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.003s sys 0.000s
I20260929 18:00:28.341846 20242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "b7d1a5a21de1419fb55434bf9207c1db"
format_stamp: "Formatted at 2026-09-29 18:00:28 on dist-test-slave-r7ph"
server_key: "3e9e69cda5a05d11168599ec1d8bbcf0"
server_key_iv: "b5f7ccc522c77c953d0c83dc12e2f144"
server_key_version: "kuduclusterkey@0"
I20260929 18:00:28.342006 20242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:28.362583 20242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:28.370321 20242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:28.370541 20242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:28.371309 20242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:00:28.372462 20242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:00:28.372538 20242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:28.372601 20242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:00:28.372630 20242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:28.383502 20242 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:36667
I20260929 18:00:28.384770 20242 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 18:00:28.389835 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 20242
I20260929 18:00:28.389983 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance
I20260929 18:00:28.390350 7640 external_mini_cluster.cc:1428] Setting key 14b443e78f8a773b3cafb3c637a196da
I20260929 18:00:28.389635 20370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:36667 every 8 connection(s)
Sep 29 18:00:28 dist-test-slave-r7ph krb5kdc[19411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704828, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:00:28.409062 20104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:28.393376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:45539 (local address 127.7.118.62:41935)
0929 18:00:28.393501 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:28.393506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:28.394044 (+ 538us) server_negotiation.cc:407] Connection header received
0929 18:00:28.395399 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:28.395402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:28.395457 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:28.395507 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:28.397247 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:28.398114 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:28.400177 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:28.400548 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:28.400641 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:28.404218 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:28.404222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:28.404224 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:28.404260 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:28.406136 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:28.406908 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:28.406912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:28.406914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:28.407006 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:28.407515 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:28.407518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:28.407520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:28.407776 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:28.407862 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:28.408713 (+ 851us) server_negotiation.cc:299] Negotiation successful
0929 18:00:28.408874 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:00:28.410874 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:28.392169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:41935 (local address 127.7.118.3:45539)
0929 18:00:28.393873 (+ 1704us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:28.393934 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:28.395141 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:28.395962 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:28.395972 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:28.396352 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:00:28.397061 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:28.397080 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:28.398315 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:28.398318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:28.400016 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:28.400030 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:28.400892 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:28.400896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:28.401117 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:28.402008 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:00:28.402032 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:00:28.403996 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:00:28.406338 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:28.406345 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:28.406367 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:28.406779 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:28.407175 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:28.407178 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:28.407180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:28.407389 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:28.407933 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:00:28.407940 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:00:28.408410 (+ 470us) client_negotiation.cc:769] Sending connection context
0929 18:00:28.409320 (+ 910us) client_negotiation.cc:240] Negotiation successful
0929 18:00:28.409709 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1456,"thread_start_us":162,"threads_started":1}
I20260929 18:00:28.412412 20371 heartbeater.cc:347] Connected to a master server at 127.7.118.62:41935
I20260929 18:00:28.412726 20371 heartbeater.cc:464] Registering TS with master...
I20260929 18:00:28.413414 20371 heartbeater.cc:511] Master 127.7.118.62:41935 requested a full tablet report, sending...
I20260929 18:00:28.414820 19887 ts_manager.cc:194] Registered new tserver with Master: b7d1a5a21de1419fb55434bf9207c1db (127.7.118.3:36667)
I20260929 18:00:28.415627 19887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.3@KRBTEST.COM'} at 127.7.118.3:45539
I20260929 18:00:28.418936 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:00:28.419893 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 19968
I20260929 18:00:28.427372 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 20107
I20260929 18:00:28.435515 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 20242
I20260929 18:00:28.444397 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 19855
I20260929 18:00:28.452458 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:41935
--webserver_interface=127.7.118.62
--webserver_port=42543
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:41935
--ranger_config_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.118.20:59261/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.118.20:59261/kms with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:00:28.577716 20374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:28.578047 20374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:28.578088 20374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:00:28.582310 20374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:00:28.582389 20374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:28.582409 20374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:28.582425 20374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:00:28.582441 20374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:00:28.588304 20374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.118.20:59261/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:41935
--ranger_config_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:41935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=42543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:28.589447 20374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:28.590517 20374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:00:28.600168 20382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:28.600199 20379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:28.600301 20374 server_base.cc:1034] running on GCE node
W20260929 18:00:28.600194 20380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:28.601125 20374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:28.602595 20374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:28.603884 20374 hybrid_clock.cc:648] HybridClock initialized: now 1790704828603855 us; error 68 us; skew 500 ppm
Sep 29 18:00:28 dist-test-slave-r7ph krb5kdc[19411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704828, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:28.607141 20374 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 18:00:28.608184 20374 webserver.cc:460] Webserver started at http://127.7.118.62:42543/ using document root <none> and password file <none>
I20260929 18:00:28.608724 20374 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:28.608804 20374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:00:28 dist-test-slave-r7ph krb5kdc[19411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704828, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for HTTP/127.7.118.20@KRBTEST.COM
I20260929 18:00:28.675539 20374 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:00:28.678354 20390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:28.679926 20374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:00:28.680065 20374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "c492e5d3289b413cb11b1e1a5b64347f"
format_stamp: "Formatted at 2026-09-29 18:00:26 on dist-test-slave-r7ph"
server_key: "d7420b61e2d9b5de3644b0c78ef2e71c"
server_key_iv: "beead2d3454543583aff7e4ff76e2b39"
server_key_version: "kuduclusterkey@0"
I20260929 18:00:28.680706 20374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:28.706454 20374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:28.710052 20374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:28.710235 20374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:28.718807 20374 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:41935
I20260929 18:00:28.718837 20442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:41935 every 8 connection(s)
I20260929 18:00:28.720073 20374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:00:28.721839 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 20374
I20260929 18:00:28.722383 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.118.20:59261/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:37903
--local_ip_for_outbound_sockets=127.7.118.1
--tserver_master_addrs=127.7.118.62:41935
--webserver_port=34459
--webserver_interface=127.7.118.1
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.118.20:59261/kms with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:00:28.724941 20443 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:00:28.727026 20443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f: Bootstrap starting.
I20260929 18:00:28.745146 20443 log.cc:824] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:00:28.752189 20443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:00:28.752662 20443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f: Bootstrap complete.
I20260929 18:00:28.757294 20443 raft_consensus.cc:348] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c492e5d3289b413cb11b1e1a5b64347f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 41935 } }
I20260929 18:00:28.757748 20443 raft_consensus.cc:729] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c492e5d3289b413cb11b1e1a5b64347f, State: Initialized, Role: FOLLOWER
I20260929 18:00:28.758333 20443 consensus_queue.cc:260] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c492e5d3289b413cb11b1e1a5b64347f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 41935 } }
I20260929 18:00:28.758486 20443 raft_consensus.cc:388] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:00:28.758576 20443 raft_consensus.cc:482] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:00:28.758697 20443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:00:28.760842 20443 raft_consensus.cc:504] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c492e5d3289b413cb11b1e1a5b64347f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 41935 } }
I20260929 18:00:28.761217 20443 leader_election.cc:302] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c492e5d3289b413cb11b1e1a5b64347f; no voters:
I20260929 18:00:28.761579 20443 leader_election.cc:288] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:00:28.761679 20447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:00:28.761904 20447 raft_consensus.cc:686] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [term 2 LEADER]: Becoming Leader. State: Replica: c492e5d3289b413cb11b1e1a5b64347f, State: Running, Role: LEADER
I20260929 18:00:28.762214 20447 consensus_queue.cc:237] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c492e5d3289b413cb11b1e1a5b64347f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 41935 } }
I20260929 18:00:28.762974 20443 sys_catalog.cc:564] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:00:28.763684 20448 sys_catalog.cc:455] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c492e5d3289b413cb11b1e1a5b64347f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c492e5d3289b413cb11b1e1a5b64347f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 41935 } } }
I20260929 18:00:28.763815 20448 sys_catalog.cc:458] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [sys.catalog]: This master's current role is: LEADER
I20260929 18:00:28.764134 20449 sys_catalog.cc:455] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c492e5d3289b413cb11b1e1a5b64347f. Latest consensus state: current_term: 2 leader_uuid: "c492e5d3289b413cb11b1e1a5b64347f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c492e5d3289b413cb11b1e1a5b64347f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 41935 } } }
I20260929 18:00:28.764271 20449 sys_catalog.cc:458] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f [sys.catalog]: This master's current role is: LEADER
I20260929 18:00:28.764564 20454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:00:28.767982 20454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:00:28.768595 20443 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 18:00:28.768813 20454 catalog_manager.cc:1259] Loaded cluster ID: 2b8af73882e04446be95d146a9afc4d7
I20260929 18:00:28.768920 20454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:00:28.774353 20454 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:00:28.776124 20454 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f: Loaded TSK: 0
I20260929 18:00:28.777987 20454 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:00:28.878064 20445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:28.878515 20445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:28.878587 20445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:00:28.884934 20445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:28.885027 20445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:28.885119 20445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 18:00:28.893551 20445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.118.20:59261/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:37903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=34459
--tserver_master_addrs=127.7.118.62:41935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:28.895085 20445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:28.896524 20445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:00:28.910382 20480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:28.910988 20483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:28.910362 20481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:28.911723 20445 server_base.cc:1034] running on GCE node
I20260929 18:00:28.912329 20445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:28.913269 20445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:28.920739 20445 hybrid_clock.cc:648] HybridClock initialized: now 1790704828920682 us; error 79 us; skew 500 ppm
Sep 29 18:00:28 dist-test-slave-r7ph krb5kdc[19411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704828, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:28.925585 20445 init.cc:377] Logged in from keytab as kudu/127.7.118.1@KRBTEST.COM (short username kudu)
I20260929 18:00:28.927501 20445 webserver.cc:460] Webserver started at http://127.7.118.1:34459/ using document root <none> and password file <none>
I20260929 18:00:28.928341 20445 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:28.928467 20445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:00:28 dist-test-slave-r7ph krb5kdc[19411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704828, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for HTTP/127.7.118.20@KRBTEST.COM
I20260929 18:00:28.971616 20445 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
I20260929 18:00:28.975008 20491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:28.977149 20445 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:00:28.977347 20445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "543cbcfe2b464c4da5209dd97a87c626"
format_stamp: "Formatted at 2026-09-29 18:00:27 on dist-test-slave-r7ph"
server_key: "020e8867a4b2aabcedbb78d5b9c86894"
server_key_iv: "5463d8f16b6d4c671434c96ffcaca951"
server_key_version: "kuduclusterkey@0"
I20260929 18:00:28.978365 20445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:29.009691 20445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:29.015720 20445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:29.016141 20445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:29.017575 20445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:00:29.019330 20445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:00:29.019682 20445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 18:00:29.019968 20445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:00:29.020236 20445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:29.033646 20445 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:37903
I20260929 18:00:29.035346 20445 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 18:00:29.044448 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 20445
I20260929 18:00:29.045105 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.118.20:59261/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:40527
--local_ip_for_outbound_sockets=127.7.118.2
--tserver_master_addrs=127.7.118.62:41935
--webserver_port=34819
--webserver_interface=127.7.118.2
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.118.20:59261/kms with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:00:29.045581 20604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:37903 every 8 connection(s)
Sep 29 18:00:29 dist-test-slave-r7ph krb5kdc[19411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704828, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:00:29.090498 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:29.047325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:60051 (local address 127.7.118.62:41935)
0929 18:00:29.049768 (+ 2443us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:29.049799 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:29.052274 (+ 2475us) server_negotiation.cc:407] Connection header received
0929 18:00:29.052711 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:29.052732 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:29.053303 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:29.053779 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:29.056196 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:29.058199 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:29.060481 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:29.061454 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:29.061703 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:29.080744 (+ 19041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:29.080753 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:29.080776 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:29.080824 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:29.083352 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:29.084284 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:29.084293 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:29.084299 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:29.084479 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:29.085119 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:29.085123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:29.085125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:29.085666 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:29.085829 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:29.088707 (+ 2878us) server_negotiation.cc:299] Negotiation successful
0929 18:00:29.089089 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2189,"thread_start_us":157,"threads_started":1}
I20260929 18:00:29.092885 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:29.042518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:41935 (local address 127.7.118.1:60051)
0929 18:00:29.050292 (+ 7774us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:29.050381 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:29.051776 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:29.054606 (+ 2830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:29.054617 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:29.055042 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:00:29.055869 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:29.055897 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:29.058493 (+ 2596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:29.058498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:29.060236 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:29.060259 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:29.062011 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:29.062015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:29.062290 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:29.078233 (+ 15943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:00:29.078291 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:00:29.080339 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:00:29.083573 (+ 3234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:29.083581 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:29.083597 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:29.084104 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:29.084606 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:29.084610 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:29.084612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:29.084830 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:29.085893 (+ 1063us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:00:29.085901 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:00:29.086321 (+ 420us) client_negotiation.cc:769] Sending connection context
0929 18:00:29.086533 (+ 212us) client_negotiation.cc:240] Negotiation successful
0929 18:00:29.087002 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7390,"thread_start_us":202,"threads_started":1}
I20260929 18:00:29.098459 20605 heartbeater.cc:347] Connected to a master server at 127.7.118.62:41935
I20260929 18:00:29.098768 20605 heartbeater.cc:464] Registering TS with master...
I20260929 18:00:29.099496 20605 heartbeater.cc:511] Master 127.7.118.62:41935 requested a full tablet report, sending...
W20260929 18:00:29.102212 20605 heartbeater.cc:644] Failed to heartbeat to 127.7.118.62:41935 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 18:00:29.102396 20605 heartbeater.cc:464] Registering TS with master...
I20260929 18:00:29.102626 20605 heartbeater.cc:511] Master 127.7.118.62:41935 requested a full tablet report, sending...
I20260929 18:00:29.103662 20605 heartbeater.cc:464] Registering TS with master...
I20260929 18:00:29.103899 20605 heartbeater.cc:511] Master 127.7.118.62:41935 requested a full tablet report, sending...
W20260929 18:00:29.104848 20605 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:00:29.248564 20608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:29.249078 20608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:29.249169 20608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:00:29.257303 20608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:29.257421 20608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:29.257532 20608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 18:00:29.269170 20608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.118.20:59261/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:40527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=34819
--tserver_master_addrs=127.7.118.62:41935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:29.270951 20608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:29.272780 20608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:00:29.288309 20615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:29.288314 20617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:29.289074 20614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:29.289212 20608 server_base.cc:1034] running on GCE node
I20260929 18:00:29.289928 20608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:29.291049 20608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:29.292371 20608 hybrid_clock.cc:648] HybridClock initialized: now 1790704829292326 us; error 70 us; skew 500 ppm
Sep 29 18:00:29 dist-test-slave-r7ph krb5kdc[19411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704829, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:29.296772 20608 init.cc:377] Logged in from keytab as kudu/127.7.118.2@KRBTEST.COM (short username kudu)
I20260929 18:00:29.298357 20608 webserver.cc:460] Webserver started at http://127.7.118.2:34819/ using document root <none> and password file <none>
I20260929 18:00:29.299316 20608 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:29.299525 20608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:00:29 dist-test-slave-r7ph krb5kdc[19411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704829, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for HTTP/127.7.118.20@KRBTEST.COM
I20260929 18:00:29.322979 20608 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260929 18:00:29.327978 20625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:29.330016 20608 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 18:00:29.330188 20608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "7d3e234f366049d586a0803cc32d2d30"
format_stamp: "Formatted at 2026-09-29 18:00:27 on dist-test-slave-r7ph"
server_key: "f9e9bd4ac9c278bb87059798d7db2bb4"
server_key_iv: "916ffc915fa997c4fe7c53a977349fb6"
server_key_version: "kuduclusterkey@0"
I20260929 18:00:29.331033 20608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:29.365701 20608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:29.369688 20608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:29.369930 20608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:29.370640 20608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:00:29.371567 20608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:00:29.371642 20608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:29.371708 20608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:00:29.371760 20608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:29.383402 20608 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:40527
I20260929 18:00:29.383419 20739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:40527 every 8 connection(s)
I20260929 18:00:29.384943 20608 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 18:00:29.394913 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 20608
I20260929 18:00:29.395457 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.118.20:59261/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:36667
--local_ip_for_outbound_sockets=127.7.118.3
--tserver_master_addrs=127.7.118.62:41935
--webserver_port=36057
--webserver_interface=127.7.118.3
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.118.20:59261/kms with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:00:29 dist-test-slave-r7ph krb5kdc[19411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704829, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:00:29.404090 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:29.388955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:46489 (local address 127.7.118.62:41935)
0929 18:00:29.389093 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:29.389098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:29.389585 (+ 487us) server_negotiation.cc:407] Connection header received
0929 18:00:29.390952 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:29.390956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:29.391016 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:29.391072 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:29.392821 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:29.393365 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:29.394973 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:29.395340 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:29.395431 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:29.399084 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:29.399092 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:29.399097 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:29.399143 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:29.401269 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:29.402134 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:29.402138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:29.402140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:29.402242 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:29.402746 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:29.402750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:29.402752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:29.402971 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:29.403094 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:29.403728 (+ 634us) server_negotiation.cc:299] Negotiation successful
0929 18:00:29.403886 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:00:29.405181 20742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:29.387697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:41935 (local address 127.7.118.2:46489)
0929 18:00:29.389396 (+ 1699us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:29.389461 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:29.390662 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:29.391516 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:29.391527 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:29.391918 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:00:29.392650 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:29.392667 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:29.393508 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:29.393512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:29.394795 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:29.394811 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:29.395677 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:29.395683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:29.395962 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:29.396787 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:00:29.396819 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:00:29.398844 (+ 2025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:00:29.401444 (+ 2600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:29.401452 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:29.401472 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:29.401989 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:29.402414 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:29.402418 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:29.402420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:29.402612 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:29.403094 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:00:29.403102 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:00:29.403441 (+ 339us) client_negotiation.cc:769] Sending connection context
0929 18:00:29.403663 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 18:00:29.404064 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1456,"thread_start_us":178,"threads_started":1}
I20260929 18:00:29.406719 20740 heartbeater.cc:347] Connected to a master server at 127.7.118.62:41935
I20260929 18:00:29.407001 20740 heartbeater.cc:464] Registering TS with master...
I20260929 18:00:29.407579 20740 heartbeater.cc:511] Master 127.7.118.62:41935 requested a full tablet report, sending...
W20260929 18:00:29.409320 20740 heartbeater.cc:644] Failed to heartbeat to 127.7.118.62:41935 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 18:00:29.409468 20740 heartbeater.cc:464] Registering TS with master...
I20260929 18:00:29.409679 20740 heartbeater.cc:511] Master 127.7.118.62:41935 requested a full tablet report, sending...
I20260929 18:00:29.410872 20740 heartbeater.cc:464] Registering TS with master...
I20260929 18:00:29.411095 20740 heartbeater.cc:511] Master 127.7.118.62:41935 requested a full tablet report, sending...
W20260929 18:00:29.412060 20740 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:00:29.559994 20743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:29.560429 20743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:29.560500 20743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:00:29.567022 20743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:29.567116 20743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:29.567206 20743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 18:00:29.575138 20743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.118.20:59261/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:36667
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=36057
--tserver_master_addrs=127.7.118.62:41935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:29.576668 20743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:29.578092 20743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:00:29.590117 20751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:29.590116 20748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:29.595145 20749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:29.596273 20743 server_base.cc:1034] running on GCE node
I20260929 18:00:29.596983 20743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:29.597946 20743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:29.599166 20743 hybrid_clock.cc:648] HybridClock initialized: now 1790704829599122 us; error 66 us; skew 500 ppm
Sep 29 18:00:29 dist-test-slave-r7ph krb5kdc[19411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704829, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:29.602897 20743 init.cc:377] Logged in from keytab as kudu/127.7.118.3@KRBTEST.COM (short username kudu)
I20260929 18:00:29.604326 20743 webserver.cc:460] Webserver started at http://127.7.118.3:36057/ using document root <none> and password file <none>
I20260929 18:00:29.605151 20743 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:29.605288 20743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:00:29 dist-test-slave-r7ph krb5kdc[19411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704829, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for HTTP/127.7.118.20@KRBTEST.COM
I20260929 18:00:29.625535 20743 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:00:29.629096 20759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:29.631072 20743 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 18:00:29.631247 20743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "b7d1a5a21de1419fb55434bf9207c1db"
format_stamp: "Formatted at 2026-09-29 18:00:28 on dist-test-slave-r7ph"
server_key: "3e9e69cda5a05d11168599ec1d8bbcf0"
server_key_iv: "b5f7ccc522c77c953d0c83dc12e2f144"
server_key_version: "kuduclusterkey@0"
I20260929 18:00:29.632061 20743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:29.662338 20743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:29.666661 20743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:29.666886 20743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:29.667654 20743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:00:29.668859 20743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:00:29.668929 20743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:29.668988 20743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:00:29.669019 20743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:29.681861 20743 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:36667
I20260929 18:00:29.681933 20879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:36667 every 8 connection(s)
I20260929 18:00:29.683364 20743 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 18:00:29.686586 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 20743
Sep 29 18:00:29 dist-test-slave-r7ph krb5kdc[19411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704829, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:00:29.701659 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:29.686468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:39381 (local address 127.7.118.62:41935)
0929 18:00:29.687127 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:29.687134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:29.687987 (+ 853us) server_negotiation.cc:407] Connection header received
0929 18:00:29.689432 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:29.689436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:29.689497 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:29.689594 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:29.691273 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:29.691818 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:29.693761 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:29.694008 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:29.694105 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:29.697269 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:29.697273 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:29.697276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:29.697310 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:29.699097 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:29.699760 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:29.699764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:29.699765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:29.699848 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:29.700304 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:29.700309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:29.700311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:29.700570 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:29.700702 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:29.701331 (+ 629us) server_negotiation.cc:299] Negotiation successful
0929 18:00:29.701472 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540}
I20260929 18:00:29.702656 20882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:29.686398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:41935 (local address 127.7.118.3:39381)
0929 18:00:29.687790 (+ 1392us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:29.687858 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:29.689088 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:29.689835 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:29.689846 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:29.690294 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:00:29.691059 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:29.691078 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:29.691968 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:29.691972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:29.693598 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:29.693611 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:29.694221 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:29.694227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:29.694507 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:29.695316 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:00:29.695342 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:00:29.697081 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:00:29.699232 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:29.699248 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:29.699279 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:29.699643 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:29.699946 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:29.699949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:29.699951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:29.700165 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:29.700692 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:00:29.700702 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:00:29.701041 (+ 339us) client_negotiation.cc:769] Sending connection context
0929 18:00:29.701217 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 18:00:29.701528 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1124,"thread_start_us":180,"threads_started":1}
I20260929 18:00:29.704514 20880 heartbeater.cc:347] Connected to a master server at 127.7.118.62:41935
I20260929 18:00:29.704799 20880 heartbeater.cc:464] Registering TS with master...
I20260929 18:00:29.705646 20880 heartbeater.cc:511] Master 127.7.118.62:41935 requested a full tablet report, sending...
I20260929 18:00:29.706843 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:29.687318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47550 (local address 127.7.118.62:41935)
0929 18:00:29.694468 (+ 7150us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:29.694475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:29.694486 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:00:29.694519 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:29.694521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:29.694558 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:29.694664 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:29.695687 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:29.696369 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:29.697769 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:29.698069 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:29.698171 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:29.700164 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:29.700169 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:29.700172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:29.700198 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:29.704242 (+ 4044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:29.705076 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:29.705080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:29.705082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:29.705186 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:29.705546 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:29.705549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:29.705551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:29.705817 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:29.706028 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:29.706499 (+ 471us) server_negotiation.cc:299] Negotiation successful
0929 18:00:29.706665 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7078,"thread_start_us":109,"threads_started":1}
I20260929 18:00:29.707284 20407 ts_manager.cc:194] Registered new tserver with Master: b7d1a5a21de1419fb55434bf9207c1db (127.7.118.3:36667)
I20260929 18:00:29.708679 20407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.3@KRBTEST.COM'} at 127.7.118.3:39381
Sep 29 18:00:30 dist-test-slave-r7ph krb5kdc[19411](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790704830, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:30.105216 20605 heartbeater.cc:464] Registering TS with master...
I20260929 18:00:30.105626 20605 heartbeater.cc:511] Master 127.7.118.62:41935 requested a full tablet report, sending...
I20260929 18:00:30.106676 20407 ts_manager.cc:194] Registered new tserver with Master: 543cbcfe2b464c4da5209dd97a87c626 (127.7.118.1:37903)
I20260929 18:00:30.107509 20407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.1@KRBTEST.COM'} at 127.7.118.1:60051
I20260929 18:00:30.412326 20740 heartbeater.cc:464] Registering TS with master...
I20260929 18:00:30.412673 20740 heartbeater.cc:511] Master 127.7.118.62:41935 requested a full tablet report, sending...
I20260929 18:00:30.413654 20407 ts_manager.cc:194] Registered new tserver with Master: 7d3e234f366049d586a0803cc32d2d30 (127.7.118.2:40527)
I20260929 18:00:30.414397 20407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.2@KRBTEST.COM'} at 127.7.118.2:46489
I20260929 18:00:30.419230 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:00:30.429960 20894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:30.421155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47556 (local address 127.7.118.62:41935)
0929 18:00:30.421428 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:30.421433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:30.421859 (+ 426us) server_negotiation.cc:407] Connection header received
0929 18:00:30.422010 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:30.422013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:30.422072 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:30.422171 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:30.422943 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:30.423515 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:30.424377 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:30.424635 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:30.424749 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:30.426303 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:30.426307 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:30.426310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:30.426344 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:30.428038 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:30.428546 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:30.428550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:30.428551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:30.428658 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:30.428974 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:30.428978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:30.428979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:30.429192 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:30.429343 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:30.429620 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 18:00:30.429762 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":106,"threads_started":1}
I20260929 18:00:30.435070 20407 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:00:30.440018 20407 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:00:30.463636 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:30.456095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:36667 (local address 127.0.0.1:50320)
0929 18:00:30.457329 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:30.457402 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:30.457621 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:30.458505 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:30.458510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:30.458536 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:00:30.458690 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:30.458699 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:30.460054 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:30.460059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:30.461511 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:30.461523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:30.462952 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:30.462956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:30.463124 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:30.463184 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 18:00:30.463329 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 18:00:30.463478 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1047,"thread_start_us":125,"threads_started":1}
I20260929 18:00:30.463637 20902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:30.456613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.7.118.3:36667)
0929 18:00:30.457067 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:30.457075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:30.457499 (+ 424us) server_negotiation.cc:407] Connection header received
0929 18:00:30.457824 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:30.457833 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:30.458079 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:30.458216 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:00:30.458824 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:30.459864 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:30.461661 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:30.462591 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:30.462678 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:30.462835 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:30.463368 (+ 533us) server_negotiation.cc:299] Negotiation successful
0929 18:00:30.463478 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":122,"threads_started":1}
I20260929 18:00:30.469426 20814 tablet_service.cc:1467] Processing CreateTablet for tablet 407ec47dbfdb49c2bb86d8656d1a79c4 (DEFAULT_TABLE table=test-table [id=277c0f7f5c7f46ab9b4fe5a3240e274d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:00:30.470707 20901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:30.456948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:40527 (local address 127.0.0.1:60270)
0929 18:00:30.462451 (+ 5503us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:30.462469 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:30.462580 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:30.465323 (+ 2743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:30.465326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:30.465347 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:00:30.465456 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:30.465463 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:30.467734 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:30.467738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:30.468881 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:30.468893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:30.470207 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:30.470210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:30.470327 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:30.470353 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:00:30.470440 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 18:00:30.470530 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5419,"thread_start_us":84,"threads_started":1}
I20260929 18:00:30.471046 20814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 407ec47dbfdb49c2bb86d8656d1a79c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:00:30.473640 20905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:30.457768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60270 (local address 127.7.118.2:40527)
0929 18:00:30.464794 (+ 7026us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:30.464802 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:30.464828 (+ 26us) server_negotiation.cc:407] Connection header received
0929 18:00:30.464909 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:30.464914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:30.465135 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:30.465617 (+ 482us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:00:30.465772 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:30.466864 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:30.469052 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:30.469871 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:30.469981 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:30.473290 (+ 3309us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:30.473349 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 18:00:30.473475 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6905,"thread_start_us":77,"threads_started":1}
I20260929 18:00:30.477807 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:30.457251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:37903 (local address 127.0.0.1:54660)
0929 18:00:30.464344 (+ 7093us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:30.464358 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:30.464443 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:30.471533 (+ 7090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:30.471536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:30.471561 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:00:30.471650 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:30.471656 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:30.473976 (+ 2320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:30.473982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:30.475128 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:30.475140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:30.477345 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:30.477349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:30.477471 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:30.477503 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 18:00:30.477561 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 18:00:30.477650 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7033,"thread_start_us":77,"threads_started":1}
I20260929 18:00:30.478919 20906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:30.458091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54660 (local address 127.7.118.1:37903)
0929 18:00:30.467180 (+ 9089us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:30.467190 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:30.467217 (+ 27us) server_negotiation.cc:407] Connection header received
0929 18:00:30.467304 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:30.467310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:30.467565 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:30.471833 (+ 4268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:00:30.472004 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:30.473100 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:30.476213 (+ 3113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:30.477102 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:30.478404 (+ 1302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:30.478581 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:30.478646 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 18:00:30.478766 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8966,"thread_start_us":160,"threads_started":1}
I20260929 18:00:30.478752 20671 tablet_service.cc:1467] Processing CreateTablet for tablet 407ec47dbfdb49c2bb86d8656d1a79c4 (DEFAULT_TABLE table=test-table [id=277c0f7f5c7f46ab9b4fe5a3240e274d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:00:30.480124 20671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 407ec47dbfdb49c2bb86d8656d1a79c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:00:30.482492 20907 tablet_bootstrap.cc:492] T 407ec47dbfdb49c2bb86d8656d1a79c4 P b7d1a5a21de1419fb55434bf9207c1db: Bootstrap starting.
I20260929 18:00:30.482308 20539 tablet_service.cc:1467] Processing CreateTablet for tablet 407ec47dbfdb49c2bb86d8656d1a79c4 (DEFAULT_TABLE table=test-table [id=277c0f7f5c7f46ab9b4fe5a3240e274d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:00:30.483500 20539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 407ec47dbfdb49c2bb86d8656d1a79c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:00:30.485460 20907 tablet_bootstrap.cc:654] T 407ec47dbfdb49c2bb86d8656d1a79c4 P b7d1a5a21de1419fb55434bf9207c1db: Neither blocks nor log segments found. Creating new log.
I20260929 18:00:30.486399 20907 log.cc:824] T 407ec47dbfdb49c2bb86d8656d1a79c4 P b7d1a5a21de1419fb55434bf9207c1db: Log is configured to *not* fsync() on all Append() calls
I20260929 18:00:30.489806 20909 tablet_bootstrap.cc:492] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30: Bootstrap starting.
I20260929 18:00:30.492450 20909 tablet_bootstrap.cc:654] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30: Neither blocks nor log segments found. Creating new log.
I20260929 18:00:30.493432 20910 tablet_bootstrap.cc:492] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 543cbcfe2b464c4da5209dd97a87c626: Bootstrap starting.
I20260929 18:00:30.493484 20909 log.cc:824] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30: Log is configured to *not* fsync() on all Append() calls
I20260929 18:00:30.496297 20909 tablet_bootstrap.cc:492] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30: No bootstrap required, opened a new log
I20260929 18:00:30.496532 20909 ts_tablet_manager.cc:1397] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 18:00:30.498275 20907 tablet_bootstrap.cc:492] T 407ec47dbfdb49c2bb86d8656d1a79c4 P b7d1a5a21de1419fb55434bf9207c1db: No bootstrap required, opened a new log
I20260929 18:00:30.498521 20907 ts_tablet_manager.cc:1397] T 407ec47dbfdb49c2bb86d8656d1a79c4 P b7d1a5a21de1419fb55434bf9207c1db: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20260929 18:00:30.500849 20909 raft_consensus.cc:348] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d1a5a21de1419fb55434bf9207c1db" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36667 } } peers { permanent_uuid: "7d3e234f366049d586a0803cc32d2d30" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 40527 } } peers { permanent_uuid: "543cbcfe2b464c4da5209dd97a87c626" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 37903 } }
I20260929 18:00:30.501158 20909 raft_consensus.cc:374] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:00:30.501211 20909 raft_consensus.cc:729] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d3e234f366049d586a0803cc32d2d30, State: Initialized, Role: FOLLOWER
I20260929 18:00:30.502036 20909 consensus_queue.cc:260] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d1a5a21de1419fb55434bf9207c1db" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36667 } } peers { permanent_uuid: "7d3e234f366049d586a0803cc32d2d30" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 40527 } } peers { permanent_uuid: "543cbcfe2b464c4da5209dd97a87c626" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 37903 } }
I20260929 18:00:30.503760 20909 ts_tablet_manager.cc:1428] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 18:00:30.504310 20740 heartbeater.cc:503] Master 127.7.118.62:41935 was elected leader, sending a full tablet report...
I20260929 18:00:30.505038 20907 raft_consensus.cc:348] T 407ec47dbfdb49c2bb86d8656d1a79c4 P b7d1a5a21de1419fb55434bf9207c1db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d1a5a21de1419fb55434bf9207c1db" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36667 } } peers { permanent_uuid: "7d3e234f366049d586a0803cc32d2d30" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 40527 } } peers { permanent_uuid: "543cbcfe2b464c4da5209dd97a87c626" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 37903 } }
I20260929 18:00:30.505347 20907 raft_consensus.cc:374] T 407ec47dbfdb49c2bb86d8656d1a79c4 P b7d1a5a21de1419fb55434bf9207c1db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:00:30.505409 20907 raft_consensus.cc:729] T 407ec47dbfdb49c2bb86d8656d1a79c4 P b7d1a5a21de1419fb55434bf9207c1db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7d1a5a21de1419fb55434bf9207c1db, State: Initialized, Role: FOLLOWER
I20260929 18:00:30.506084 20907 consensus_queue.cc:260] T 407ec47dbfdb49c2bb86d8656d1a79c4 P b7d1a5a21de1419fb55434bf9207c1db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d1a5a21de1419fb55434bf9207c1db" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36667 } } peers { permanent_uuid: "7d3e234f366049d586a0803cc32d2d30" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 40527 } } peers { permanent_uuid: "543cbcfe2b464c4da5209dd97a87c626" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 37903 } }
I20260929 18:00:30.507309 20907 ts_tablet_manager.cc:1428] T 407ec47dbfdb49c2bb86d8656d1a79c4 P b7d1a5a21de1419fb55434bf9207c1db: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260929 18:00:30.508369 20910 tablet_bootstrap.cc:654] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 543cbcfe2b464c4da5209dd97a87c626: Neither blocks nor log segments found. Creating new log.
I20260929 18:00:30.508790 20880 heartbeater.cc:503] Master 127.7.118.62:41935 was elected leader, sending a full tablet report...
I20260929 18:00:30.512198 20910 log.cc:824] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 543cbcfe2b464c4da5209dd97a87c626: Log is configured to *not* fsync() on all Append() calls
I20260929 18:00:30.518550 20910 tablet_bootstrap.cc:492] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 543cbcfe2b464c4da5209dd97a87c626: No bootstrap required, opened a new log
I20260929 18:00:30.518951 20910 ts_tablet_manager.cc:1397] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 543cbcfe2b464c4da5209dd97a87c626: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.000s
I20260929 18:00:30.523697 20910 raft_consensus.cc:348] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 543cbcfe2b464c4da5209dd97a87c626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d1a5a21de1419fb55434bf9207c1db" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36667 } } peers { permanent_uuid: "7d3e234f366049d586a0803cc32d2d30" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 40527 } } peers { permanent_uuid: "543cbcfe2b464c4da5209dd97a87c626" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 37903 } }
I20260929 18:00:30.524168 20910 raft_consensus.cc:374] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 543cbcfe2b464c4da5209dd97a87c626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:00:30.524307 20910 raft_consensus.cc:729] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 543cbcfe2b464c4da5209dd97a87c626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 543cbcfe2b464c4da5209dd97a87c626, State: Initialized, Role: FOLLOWER
I20260929 18:00:30.524911 20910 consensus_queue.cc:260] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 543cbcfe2b464c4da5209dd97a87c626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d1a5a21de1419fb55434bf9207c1db" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36667 } } peers { permanent_uuid: "7d3e234f366049d586a0803cc32d2d30" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 40527 } } peers { permanent_uuid: "543cbcfe2b464c4da5209dd97a87c626" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 37903 } }
I20260929 18:00:30.526333 20910 ts_tablet_manager.cc:1428] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 543cbcfe2b464c4da5209dd97a87c626: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 18:00:30.526829 20605 heartbeater.cc:503] Master 127.7.118.62:41935 was elected leader, sending a full tablet report...
I20260929 18:00:30.527921 20912 raft_consensus.cc:482] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:00:30.528119 20912 raft_consensus.cc:504] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d1a5a21de1419fb55434bf9207c1db" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36667 } } peers { permanent_uuid: "7d3e234f366049d586a0803cc32d2d30" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 40527 } } peers { permanent_uuid: "543cbcfe2b464c4da5209dd97a87c626" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 37903 } }
I20260929 18:00:30.529755 20912 leader_election.cc:288] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7d1a5a21de1419fb55434bf9207c1db (127.7.118.3:36667), 543cbcfe2b464c4da5209dd97a87c626 (127.7.118.1:37903)
I20260929 18:00:30.536846 20902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:30.530380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:56497 (local address 127.7.118.3:36667)
0929 18:00:30.530529 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:30.530534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:30.531935 (+ 1401us) server_negotiation.cc:407] Connection header received
0929 18:00:30.532033 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:30.532036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:30.532096 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:30.532178 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:00:30.532631 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:30.533465 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:30.535325 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:30.535964 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:30.536055 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:30.536130 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:30.536543 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 18:00:30.536654 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 18:00:30.537659 20917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:30.530224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:36667 (local address 127.7.118.2:56497)
0929 18:00:30.531798 (+ 1574us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:30.531814 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:30.531939 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:30.532317 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:30.532320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:30.532341 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:00:30.532486 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:30.532493 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:30.533644 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:30.533647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:30.535164 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:30.535175 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:30.536248 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:30.536251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:30.536358 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:30.536382 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:00:30.537348 (+ 966us) client_negotiation.cc:240] Negotiation successful
0929 18:00:30.537460 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1498,"thread_start_us":136,"threads_started":1}
Sep 29 18:00:30 dist-test-slave-r7ph krb5kdc[19411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704830, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for HTTP/127.7.118.20@KRBTEST.COM
I20260929 18:00:30.538827 20834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407ec47dbfdb49c2bb86d8656d1a79c4" candidate_uuid: "7d3e234f366049d586a0803cc32d2d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7d1a5a21de1419fb55434bf9207c1db" is_pre_election: true
I20260929 18:00:30.539222 20834 raft_consensus.cc:2454] T 407ec47dbfdb49c2bb86d8656d1a79c4 P b7d1a5a21de1419fb55434bf9207c1db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d3e234f366049d586a0803cc32d2d30 in term 0.
I20260929 18:00:30.539839 20629 leader_election.cc:302] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d3e234f366049d586a0803cc32d2d30, b7d1a5a21de1419fb55434bf9207c1db; no voters:
I20260929 18:00:30.540246 20912 raft_consensus.cc:2781] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:00:30.540337 20912 raft_consensus.cc:482] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:00:30.540385 20912 raft_consensus.cc:3037] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:00:30.541476 20906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:30.530965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:51581 (local address 127.7.118.1:37903)
0929 18:00:30.531072 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:30.531076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:30.532231 (+ 1155us) server_negotiation.cc:407] Connection header received
0929 18:00:30.532393 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:30.532396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:30.532442 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:30.532486 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:00:30.533527 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:30.534391 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:30.536239 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:30.536898 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:30.536978 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:30.537037 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:30.541213 (+ 4176us) server_negotiation.cc:299] Negotiation successful
0929 18:00:30.541342 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 18:00:30.541778 20606 tablet.cc:2369] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 543cbcfe2b464c4da5209dd97a87c626: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:00:30.541846 20912 raft_consensus.cc:504] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d1a5a21de1419fb55434bf9207c1db" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36667 } } peers { permanent_uuid: "7d3e234f366049d586a0803cc32d2d30" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 40527 } } peers { permanent_uuid: "543cbcfe2b464c4da5209dd97a87c626" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 37903 } }
I20260929 18:00:30.542292 20912 leader_election.cc:288] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30 [CANDIDATE]: Term 1 election: Requested vote from peers b7d1a5a21de1419fb55434bf9207c1db (127.7.118.3:36667), 543cbcfe2b464c4da5209dd97a87c626 (127.7.118.1:37903)
I20260929 18:00:30.543418 20834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407ec47dbfdb49c2bb86d8656d1a79c4" candidate_uuid: "7d3e234f366049d586a0803cc32d2d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7d1a5a21de1419fb55434bf9207c1db"
I20260929 18:00:30.543577 20834 raft_consensus.cc:3037] T 407ec47dbfdb49c2bb86d8656d1a79c4 P b7d1a5a21de1419fb55434bf9207c1db [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:00:30.545179 20834 raft_consensus.cc:2454] T 407ec47dbfdb49c2bb86d8656d1a79c4 P b7d1a5a21de1419fb55434bf9207c1db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d3e234f366049d586a0803cc32d2d30 in term 1.
I20260929 18:00:30.545686 20629 leader_election.cc:302] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d3e234f366049d586a0803cc32d2d30, b7d1a5a21de1419fb55434bf9207c1db; no voters:
I20260929 18:00:30.545897 20912 raft_consensus.cc:2781] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:00:30.546068 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:30.530638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:37903 (local address 127.7.118.2:51581)
0929 18:00:30.532157 (+ 1519us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:30.532169 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:30.532287 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:30.532641 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:30.532643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:30.532659 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:00:30.533378 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:30.533391 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:30.534754 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:30.534757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:30.536062 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:30.536072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:30.540890 (+ 4818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:30.540895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:30.541042 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:30.541068 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:00:30.545670 (+ 4602us) client_negotiation.cc:240] Negotiation successful
0929 18:00:30.545802 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1464,"mutex_wait_us":3,"thread_start_us":77,"threads_started":1}
I20260929 18:00:30.546133 20912 raft_consensus.cc:686] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30 [term 1 LEADER]: Becoming Leader. State: Replica: 7d3e234f366049d586a0803cc32d2d30, State: Running, Role: LEADER
I20260929 18:00:30.546598 20912 consensus_queue.cc:237] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d1a5a21de1419fb55434bf9207c1db" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36667 } } peers { permanent_uuid: "7d3e234f366049d586a0803cc32d2d30" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 40527 } } peers { permanent_uuid: "543cbcfe2b464c4da5209dd97a87c626" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 37903 } }
I20260929 18:00:30.547209 20557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407ec47dbfdb49c2bb86d8656d1a79c4" candidate_uuid: "7d3e234f366049d586a0803cc32d2d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "543cbcfe2b464c4da5209dd97a87c626" is_pre_election: true
I20260929 18:00:30.547559 20557 raft_consensus.cc:2454] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 543cbcfe2b464c4da5209dd97a87c626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d3e234f366049d586a0803cc32d2d30 in term 0.
I20260929 18:00:30.547909 20558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407ec47dbfdb49c2bb86d8656d1a79c4" candidate_uuid: "7d3e234f366049d586a0803cc32d2d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "543cbcfe2b464c4da5209dd97a87c626"
I20260929 18:00:30.548007 20558 raft_consensus.cc:3037] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 543cbcfe2b464c4da5209dd97a87c626 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:00:30.549393 20558 raft_consensus.cc:2454] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 543cbcfe2b464c4da5209dd97a87c626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d3e234f366049d586a0803cc32d2d30 in term 1.
I20260929 18:00:30.550668 20407 catalog_manager.cc:5495] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d3e234f366049d586a0803cc32d2d30 (127.7.118.2). New cstate: current_term: 1 leader_uuid: "7d3e234f366049d586a0803cc32d2d30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d1a5a21de1419fb55434bf9207c1db" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d3e234f366049d586a0803cc32d2d30" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 40527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "543cbcfe2b464c4da5209dd97a87c626" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 37903 } health_report { overall_health: UNKNOWN } } }
I20260929 18:00:30.572311 20905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:30.566895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60274 (local address 127.7.118.2:40527)
0929 18:00:30.567043 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:30.567049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:30.567065 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:00:30.567165 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:30.567168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:30.567246 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:30.567379 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:00:30.567696 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:30.568420 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:30.569806 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:30.570137 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:30.570270 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:30.570742 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:00:30.571404 (+ 662us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:00:30.571523 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:30.572031 (+ 508us) server_negotiation.cc:299] Negotiation successful
0929 18:00:30.572107 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:00:30.580977 20558 raft_consensus.cc:1261] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 543cbcfe2b464c4da5209dd97a87c626 [term 1 FOLLOWER]: Refusing update from remote peer 7d3e234f366049d586a0803cc32d2d30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:00:30.582151 20918 consensus_queue.cc:1035] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543cbcfe2b464c4da5209dd97a87c626" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:00:30.584863 20834 raft_consensus.cc:1261] T 407ec47dbfdb49c2bb86d8656d1a79c4 P b7d1a5a21de1419fb55434bf9207c1db [term 1 FOLLOWER]: Refusing update from remote peer 7d3e234f366049d586a0803cc32d2d30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:00:30.585778 20918 consensus_queue.cc:1035] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7d1a5a21de1419fb55434bf9207c1db" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 36667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:00:30.599700 20921 mvcc.cc:205] Tried to move back new op lower bound from 7334726986049236992 to 7334726985923772416. Current Snapshot: MvccSnapshot[applied={T|T < 7334726986049236992}]
I20260929 18:00:30.600888 20922 mvcc.cc:205] Tried to move back new op lower bound from 7334726986049236992 to 7334726985923772416. Current Snapshot: MvccSnapshot[applied={T|T < 7334726986049236992}]
I20260929 18:00:30.603102 20924 mvcc.cc:205] Tried to move back new op lower bound from 7334726986049236992 to 7334726985923772416. Current Snapshot: MvccSnapshot[applied={T|T < 7334726986049236992}]
I20260929 18:00:30.611043 20406 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:00:30.611258 20406 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:00:30.614305 20406 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c492e5d3289b413cb11b1e1a5b64347f: Sending DeleteTablet for 3 replicas of tablet 407ec47dbfdb49c2bb86d8656d1a79c4
I20260929 18:00:30.615268 20814 tablet_service.cc:1514] Processing DeleteTablet for tablet 407ec47dbfdb49c2bb86d8656d1a79c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:00:30 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:50320
I20260929 18:00:30.616173 20539 tablet_service.cc:1514] Processing DeleteTablet for tablet 407ec47dbfdb49c2bb86d8656d1a79c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:00:30 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:54660
I20260929 18:00:30.616173 20671 tablet_service.cc:1514] Processing DeleteTablet for tablet 407ec47dbfdb49c2bb86d8656d1a79c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:00:30 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:60270
I20260929 18:00:30.617151 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 20445
I20260929 18:00:30.620570 20932 tablet_replica.cc:330] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30: stopping tablet replica
I20260929 18:00:30.620932 20932 raft_consensus.cc:2229] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:00:30.621382 20932 raft_consensus.cc:2258] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:00:30.628310 20932 ts_tablet_manager.cc:1905] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:00:30.631774 20932 ts_tablet_manager.cc:1918] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260929 18:00:30.632758 20394 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.118.1:37903 (error 108)
I20260929 18:00:30.633893 20932 log.cc:1196] T 407ec47dbfdb49c2bb86d8656d1a79c4 P 7d3e234f366049d586a0803cc32d2d30: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-1/wal/wals/407ec47dbfdb49c2bb86d8656d1a79c4
I20260929 18:00:30.634308 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 20608
I20260929 18:00:30.634513 20931 tablet_replica.cc:330] T 407ec47dbfdb49c2bb86d8656d1a79c4 P b7d1a5a21de1419fb55434bf9207c1db: stopping tablet replica
I20260929 18:00:30.634992 20931 raft_consensus.cc:2229] T 407ec47dbfdb49c2bb86d8656d1a79c4 P b7d1a5a21de1419fb55434bf9207c1db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:00:30.635382 20931 raft_consensus.cc:2258] T 407ec47dbfdb49c2bb86d8656d1a79c4 P b7d1a5a21de1419fb55434bf9207c1db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:00:30.637055 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:30.636714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:37903 (local address 127.0.0.1:54674)
0929 18:00:30.636872 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:30.636956 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.1:37903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 18:00:30.637240 20931 ts_tablet_manager.cc:1905] T 407ec47dbfdb49c2bb86d8656d1a79c4 P b7d1a5a21de1419fb55434bf9207c1db: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 18:00:30.637662 20394 catalog_manager.cc:4584] TS 543cbcfe2b464c4da5209dd97a87c626 (127.7.118.1:37903): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 407ec47dbfdb49c2bb86d8656d1a79c4: Network error: Client connection negotiation failed: client connection to 127.7.118.1:37903: connect: Connection refused (error 111)
I20260929 18:00:30.641263 20931 ts_tablet_manager.cc:1918] T 407ec47dbfdb49c2bb86d8656d1a79c4 P b7d1a5a21de1419fb55434bf9207c1db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:00:30.641402 20931 log.cc:1196] T 407ec47dbfdb49c2bb86d8656d1a79c4 P b7d1a5a21de1419fb55434bf9207c1db: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704657566865-7640-0/minicluster-data/ts-2/wal/wals/407ec47dbfdb49c2bb86d8656d1a79c4
I20260929 18:00:30.641762 20931 ts_tablet_manager.cc:1939] T 407ec47dbfdb49c2bb86d8656d1a79c4 P b7d1a5a21de1419fb55434bf9207c1db: Deleting consensus metadata
I20260929 18:00:30.643486 20394 catalog_manager.cc:4841] TS b7d1a5a21de1419fb55434bf9207c1db (127.7.118.3:36667): tablet 407ec47dbfdb49c2bb86d8656d1a79c4 (table test-table [id=277c0f7f5c7f46ab9b4fe5a3240e274d]) successfully deleted
I20260929 18:00:30.644140 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 20743
I20260929 18:00:30.645095 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:30.644355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:40527 (local address 127.0.0.1:60282)
0929 18:00:30.644949 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:30.645006 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.2:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506}
I20260929 18:00:30.652024 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 20374
I20260929 18:00:30.661469 7640 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 18:00:31.007750 7640 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 18:00:31.007920 7640 mini_ranger.cc:66] Stopping Ranger...
I20260929 18:00:31.354219 7640 mini_ranger.cc:68] Stopped Ranger
2026-09-29 18:00:31.354 UTC [19441] LOG: received smart shutdown request
2026-09-29 18:00:31.359 UTC [19441] LOG: background worker "logical replication launcher" (PID 19450) exited with exit code 1
2026-09-29 18:00:31.359 UTC [19445] LOG: shutting down
2026-09-29 18:00:31.549 UTC [19441] LOG: database system is shut down
2026-09-29T18:00:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55295 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:00:31 dist-test-slave-r7ph krb5kdc[20943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:00:31 dist-test-slave-r7ph krb5kdc[20943](info): set up 2 sockets
Sep 29 18:00:31 dist-test-slave-r7ph krb5kdc[20943](info): commencing operation
krb5kdc: starting...
W20260929 18:00:32.730386 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 18:00:32 dist-test-slave-r7ph krb5kdc[20943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:00:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:00:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 18:00:32.864252 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:45759
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:41041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:45759,127.7.118.61:39499,127.7.118.60:42137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:00:32.985921 20956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:32.986364 20956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:32.986630 20956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:00:32.991459 20956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:00:32.991580 20956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:32.991621 20956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:32.991663 20956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:00:32.991706 20956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:00:32.998070 20956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:41041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:45759,127.7.118.61:39499,127.7.118.60:42137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:45759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:32.999384 20956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:33.000556 20956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:00:33.011456 20961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:33.011722 20964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:33.011456 20962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:33.011484 20956 server_base.cc:1034] running on GCE node
I20260929 18:00:33.012390 20956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:33.013724 20956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:33.014966 20956 hybrid_clock.cc:648] HybridClock initialized: now 1790704833014903 us; error 77 us; skew 500 ppm
Sep 29 18:00:33 dist-test-slave-r7ph krb5kdc[20943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704833, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:33.018932 20956 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 18:00:33.019963 20956 webserver.cc:460] Webserver started at http://127.7.118.62:38627/ using document root <none> and password file <none>
I20260929 18:00:33.020581 20956 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:33.020650 20956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:00:33.020833 20956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:00:33.023445 20956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "15ee513781524ed6a3a95773165d7402"
format_stamp: "Formatted at 2026-09-29 18:00:33 on dist-test-slave-r7ph"
server_key: "76d5aea0e4ab0681b5b027317f7c3f2a"
server_key_iv: "b30e763de83259a3d87697489cdc9ca8"
server_key_version: "encryptionkey@0"
I20260929 18:00:33.024120 20956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "15ee513781524ed6a3a95773165d7402"
format_stamp: "Formatted at 2026-09-29 18:00:33 on dist-test-slave-r7ph"
server_key: "76d5aea0e4ab0681b5b027317f7c3f2a"
server_key_iv: "b30e763de83259a3d87697489cdc9ca8"
server_key_version: "encryptionkey@0"
I20260929 18:00:33.028761 20956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:00:33.031710 20971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:33.033170 20956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 18:00:33.033304 20956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "15ee513781524ed6a3a95773165d7402"
format_stamp: "Formatted at 2026-09-29 18:00:33 on dist-test-slave-r7ph"
server_key: "76d5aea0e4ab0681b5b027317f7c3f2a"
server_key_iv: "b30e763de83259a3d87697489cdc9ca8"
server_key_version: "encryptionkey@0"
I20260929 18:00:33.033419 20956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:33.048522 20956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:33.052330 20956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:33.052511 20956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:33.060590 20956 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:45759
I20260929 18:00:33.060619 21023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:45759 every 8 connection(s)
I20260929 18:00:33.061700 20956 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:00:33.062141 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 20956
I20260929 18:00:33.062345 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:00:33.062664 7640 external_mini_cluster.cc:1428] Setting key 5cff848ace812cab9f9a0d1b55561500
I20260929 18:00:33.065491 21024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:00:33.069978 21024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45759 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.118.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.61@KRBTEST.COM" created.
Sep 29 18:00:33 dist-test-slave-r7ph krb5kdc[20943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704833, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:00:33.088258 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.072926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51660 (local address 127.7.118.62:45759)
0929 18:00:33.073496 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:33.073517 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:33.073689 (+ 172us) server_negotiation.cc:407] Connection header received
0929 18:00:33.075253 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:33.075258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:33.075510 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:33.075719 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:33.076813 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.077970 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.079222 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.079644 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.079835 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:33.082702 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:33.082709 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:33.082722 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:33.082746 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:33.084664 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.085218 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.085229 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.085231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.085357 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.085711 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.085715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.085718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.086157 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:33.086344 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:33.086713 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 18:00:33.086995 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"spinlock_wait_cycles":19200,"thread_start_us":226,"threads_started":1}
I20260929 18:00:33.088255 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.072926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45759 (local address 127.0.0.1:51660)
0929 18:00:33.073477 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.073533 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:33.074830 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:33.075753 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:33.075762 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:33.076324 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:00:33.076639 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:33.076654 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.078131 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.078135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:33.079068 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:33.079081 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.079823 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.079827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:33.079939 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:33.080716 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:00:33.080735 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:00:33.082500 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:00:33.084810 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:33.084820 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:33.084833 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:33.085090 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:33.085456 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:33.085461 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:33.085463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:33.085598 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:33.086298 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:00:33.086305 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:00:33.086459 (+ 154us) client_negotiation.cc:769] Sending connection context
0929 18:00:33.086638 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 18:00:33.087053 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":110,"thread_start_us":171,"threads_started":1}
I20260929 18:00:33.091428 21024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.61" port: 39499 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:00:33.092268 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.091967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:39499 (local address 127.0.0.1:53378)
0929 18:00:33.092117 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.092192 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.61:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 18:00:33.092541 20975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.118.61:39499: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.61.keytab.
Entry for principal kudu/127.7.118.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.61.keytab.
I20260929 18:00:33.093693 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.093360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:39499 (local address 127.0.0.1:53384)
0929 18:00:33.093524 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.093596 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.61:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 18:00:33.093879 21024 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.61:39499: Network error: Client connection negotiation failed: client connection to 127.7.118.61:39499: connect: Connection refused (error 111)
I20260929 18:00:33.094441 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.61:39499
--webserver_interface=127.7.118.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.61.keytab
--principal=kudu/127.7.118.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:41041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:45759,127.7.118.61:39499,127.7.118.60:42137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:00:33.143227 21024 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.61" port: 39499 } attempt: 1
I20260929 18:00:33.144213 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.143846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:39499 (local address 127.0.0.1:53390)
0929 18:00:33.144026 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.144105 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.61:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 18:00:33.145048 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.144821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:39499 (local address 127.0.0.1:53404)
0929 18:00:33.144929 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.144983 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.61:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 18:00:33.145252 21024 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.61:39499: Network error: Client connection negotiation failed: client connection to 127.7.118.61:39499: connect: Connection refused (error 111)
I20260929 18:00:33.213472 21024 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.61" port: 39499 } attempt: 2
W20260929 18:00:33.214063 21031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:33.214499 21031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:33.214597 21031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 18:00:33.214604 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.214123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:39499 (local address 127.0.0.1:53406)
0929 18:00:33.214375 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.214439 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.61:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 18:00:33.215303 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.215102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:39499 (local address 127.0.0.1:53420)
0929 18:00:33.215203 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.215250 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.61:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 18:00:33.215487 21024 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.61:39499: Network error: Client connection negotiation failed: client connection to 127.7.118.61:39499: connect: Connection refused (error 111)
W20260929 18:00:33.219394 21031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:00:33.219498 21031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:33.219547 21031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:33.219583 21031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:00:33.219616 21031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:00:33.225390 21031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:41041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:45759,127.7.118.61:39499,127.7.118.60:42137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.61.keytab
--principal=kudu/127.7.118.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.61:39499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.118.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:33.226706 21031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:33.227777 21031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:00:33.237236 21036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:33.237200 21037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:33.237267 21039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:33.237380 21031 server_base.cc:1034] running on GCE node
I20260929 18:00:33.238286 21031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:33.239535 21031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:33.240769 21031 hybrid_clock.cc:648] HybridClock initialized: now 1790704833240745 us; error 38 us; skew 500 ppm
Sep 29 18:00:33 dist-test-slave-r7ph krb5kdc[20943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704833, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:33.244102 21031 init.cc:377] Logged in from keytab as kudu/127.7.118.61@KRBTEST.COM (short username kudu)
I20260929 18:00:33.245107 21031 webserver.cc:460] Webserver started at http://127.7.118.61:40047/ using document root <none> and password file <none>
I20260929 18:00:33.245734 21031 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:33.245801 21031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:00:33.246025 21031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:00:33.248279 21031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-1/data/instance:
uuid: "d0e4d8aafab44e028689c0cbfc49f906"
format_stamp: "Formatted at 2026-09-29 18:00:33 on dist-test-slave-r7ph"
server_key: "5397b5149b84d835b694954313564ca8"
server_key_iv: "28369e7ce6e737ddeef901ac056dce92"
server_key_version: "encryptionkey@0"
I20260929 18:00:33.249068 21031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-1/wal/instance:
uuid: "d0e4d8aafab44e028689c0cbfc49f906"
format_stamp: "Formatted at 2026-09-29 18:00:33 on dist-test-slave-r7ph"
server_key: "5397b5149b84d835b694954313564ca8"
server_key_iv: "28369e7ce6e737ddeef901ac056dce92"
server_key_version: "encryptionkey@0"
I20260929 18:00:33.253674 21031 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 18:00:33.256986 21046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:33.258713 21031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:00:33.258896 21031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-1/wal
uuid: "d0e4d8aafab44e028689c0cbfc49f906"
format_stamp: "Formatted at 2026-09-29 18:00:33 on dist-test-slave-r7ph"
server_key: "5397b5149b84d835b694954313564ca8"
server_key_iv: "28369e7ce6e737ddeef901ac056dce92"
server_key_version: "encryptionkey@0"
I20260929 18:00:33.259052 21031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:33.287525 21031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:33.291369 21031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:33.291638 21031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:33.299749 21031 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.61:39499
I20260929 18:00:33.299814 21098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.61:39499 every 8 connection(s)
I20260929 18:00:33.300891 21031 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-1/data/info.pb
I20260929 18:00:33.301972 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 21031
I20260929 18:00:33.302121 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-1/wal/instance
I20260929 18:00:33.302449 7640 external_mini_cluster.cc:1428] Setting key 79bd9f3eb1aef21f9cbebf69397c6682
I20260929 18:00:33.304373 21099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:00:33.306645 21024 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.61" port: 39499 } attempt: 3
WARNING: no policy specified for kudu/127.7.118.60@KRBTEST.COM; defaulting to no policy
I20260929 18:00:33.310564 21099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45759 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.60@KRBTEST.COM" created.
Sep 29 18:00:33 dist-test-slave-r7ph krb5kdc[20943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704833, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for kudu/127.7.118.61@KRBTEST.COM
Sep 29 18:00:33 dist-test-slave-r7ph krb5kdc[20943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704833, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.61@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:00:33.322748 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.307276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:39499 (local address 127.0.0.1:53428)
0929 18:00:33.307512 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.307529 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:33.307632 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:33.310509 (+ 2877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:33.310512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:33.310683 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:00:33.310819 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:33.310826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.312735 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.312739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:33.313566 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:33.313580 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.314693 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.314700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:33.314857 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:33.315475 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:00:33.315488 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:00:33.316970 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:00:33.320095 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:33.320101 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:33.320106 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:33.320384 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:33.320862 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:33.320865 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:33.320867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:33.320980 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:33.321944 (+ 964us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:00:33.321948 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:00:33.322107 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 18:00:33.322242 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 18:00:33.322494 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260929 18:00:33.324738 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.308123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53428 (local address 127.7.118.61:39499)
0929 18:00:33.308605 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:33.308627 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:33.308678 (+ 51us) server_negotiation.cc:407] Connection header received
0929 18:00:33.309581 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:33.309608 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:33.310088 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:33.310529 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:33.311011 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.312563 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.313700 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.314538 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.314737 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:33.317234 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:33.317240 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:33.317252 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:33.317282 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:33.319937 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.320551 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.320558 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.320566 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.320723 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.321096 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.321100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.321102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.321766 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:33.321916 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:33.323179 (+ 1263us) server_negotiation.cc:299] Negotiation successful
0929 18:00:33.323542 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":315,"thread_start_us":159,"threads_started":1}
I20260929 18:00:33.324862 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.312904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51670 (local address 127.7.118.62:45759)
0929 18:00:33.313040 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:33.313044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:33.313386 (+ 342us) server_negotiation.cc:407] Connection header received
0929 18:00:33.313603 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:33.313609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:33.313673 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:33.313723 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:33.314745 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.316079 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.317663 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.318027 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.318171 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:33.321037 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:33.321062 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:33.321068 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:33.321099 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:33.322849 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.323468 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.323472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.323473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.323552 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.323889 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.323892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.323893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.324089 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:33.324230 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:33.324589 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 18:00:33.324721 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:00:33.324893 21103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.312810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45759 (local address 127.0.0.1:51670)
0929 18:00:33.313250 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.313309 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:33.313482 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:33.313891 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:33.313898 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:33.314424 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:00:33.314612 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:33.314621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.316240 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.316244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:33.317448 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:33.317475 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.318174 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.318179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:33.318632 (+ 453us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:33.319166 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:00:33.319178 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:00:33.320882 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:00:33.322979 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:33.322993 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:33.323000 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:33.323300 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:33.323655 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:33.323657 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:33.323659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:33.323751 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:33.324213 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:00:33.324220 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:00:33.324353 (+ 133us) client_negotiation.cc:769] Sending connection context
0929 18:00:33.324499 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 18:00:33.324687 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":151,"threads_started":1}
I20260929 18:00:33.326817 21099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.61" port: 39499 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:00:33.327716 21024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:00:33.328513 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.328250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42137 (local address 127.0.0.1:49060)
0929 18:00:33.328377 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.328442 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 18:00:33.329324 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.329036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42137 (local address 127.0.0.1:49068)
0929 18:00:33.329203 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.329263 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260929 18:00:33.329514 21024 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.60:42137: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42137: connect: Connection refused (error 111)
Sep 29 18:00:33 dist-test-slave-r7ph krb5kdc[20943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704833, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.61@KRBTEST.COM for kudu/127.7.118.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.60.keytab.
Entry for principal kudu/127.7.118.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.60.keytab.
I20260929 18:00:33.336241 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.60:42137
--webserver_interface=127.7.118.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.60.keytab
--principal=kudu/127.7.118.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:41041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:45759,127.7.118.61:39499,127.7.118.60:42137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:00:33.336968 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.327493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53430 (local address 127.7.118.61:39499)
0929 18:00:33.327753 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:33.327760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:33.327945 (+ 185us) server_negotiation.cc:407] Connection header received
0929 18:00:33.328031 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:33.328035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:33.328087 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:33.328174 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:33.328650 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.329347 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.330364 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.330628 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.330742 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:33.333182 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:33.333186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:33.333188 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:33.333213 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:33.335010 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.335561 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.335564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.335566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.335655 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.335959 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.335960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.335962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.336147 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:33.336218 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:33.336651 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 18:00:33.336842 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260929 18:00:33.336974 21103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.327449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:39499 (local address 127.0.0.1:53430)
0929 18:00:33.327812 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.327823 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:33.327952 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:33.328183 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:33.328186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:33.328411 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:00:33.328530 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:33.328537 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.329464 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.329467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:33.330204 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:33.330214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.330742 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.330745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:33.330838 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:33.331480 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:00:33.331503 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:00:33.332999 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:00:33.335145 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:33.335148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:33.335151 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:33.335441 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:33.335759 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:33.335762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:33.335764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:33.335868 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:33.336246 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:00:33.336250 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:00:33.336370 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 18:00:33.336489 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 18:00:33.336692 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326}
I20260929 18:00:33.338436 21099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:00:33.339291 21103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.338978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42137 (local address 127.0.0.1:49082)
0929 18:00:33.339129 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.339220 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 18:00:33.339594 21048 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42137: connect: Connection refused (error 111)
I20260929 18:00:33.340777 21103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.340459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42137 (local address 127.0.0.1:49098)
0929 18:00:33.340603 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.340678 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 18:00:33.341012 21099 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.60:42137: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42137: connect: Connection refused (error 111)
I20260929 18:00:33.360726 21024 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } attempt: 1
I20260929 18:00:33.361770 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.361391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42137 (local address 127.0.0.1:49114)
0929 18:00:33.361595 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.361674 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 18:00:33.362859 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.362595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42137 (local address 127.0.0.1:49130)
0929 18:00:33.362717 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.362778 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 18:00:33.363138 21024 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.60:42137: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42137: connect: Connection refused (error 111)
I20260929 18:00:33.390349 21099 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } attempt: 1
I20260929 18:00:33.391497 21103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.391143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42137 (local address 127.0.0.1:49144)
0929 18:00:33.391349 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.391416 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 18:00:33.392334 21103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.392118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42137 (local address 127.0.0.1:49160)
0929 18:00:33.392218 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.392275 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 18:00:33.392616 21099 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.60:42137: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42137: connect: Connection refused (error 111)
I20260929 18:00:33.438338 21024 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } attempt: 2
I20260929 18:00:33.439229 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.438873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42137 (local address 127.0.0.1:49166)
0929 18:00:33.439047 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.439124 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 18:00:33.440039 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.439788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42137 (local address 127.0.0.1:49168)
0929 18:00:33.439908 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.439965 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 18:00:33.440236 21024 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.60:42137: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42137: connect: Connection refused (error 111)
W20260929 18:00:33.459997 21105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:33.460330 21105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:33.460374 21105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 18:00:33.460812 21099 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } attempt: 2
I20260929 18:00:33.461773 21103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.461440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42137 (local address 127.0.0.1:49176)
0929 18:00:33.461619 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.461687 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 18:00:33.462652 21103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.462392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42137 (local address 127.0.0.1:49178)
0929 18:00:33.462509 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.462564 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 18:00:33.462803 21099 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.60:42137: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42137: connect: Connection refused (error 111)
W20260929 18:00:33.465471 21105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:00:33.465575 21105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:33.465622 21105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:33.465680 21105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:00:33.465718 21105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:00:33.471891 21105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:41041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:45759,127.7.118.61:39499,127.7.118.60:42137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.60.keytab
--principal=kudu/127.7.118.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.60:42137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.118.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:33.473204 21105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:33.474440 21105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:00:33.483860 21114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:33.483968 21105 server_base.cc:1034] running on GCE node
W20260929 18:00:33.483808 21112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:33.484081 21111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:33.484824 21105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:33.486096 21105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:33.487329 21105 hybrid_clock.cc:648] HybridClock initialized: now 1790704833487289 us; error 55 us; skew 500 ppm
Sep 29 18:00:33 dist-test-slave-r7ph krb5kdc[20943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704833, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:33.490640 21105 init.cc:377] Logged in from keytab as kudu/127.7.118.60@KRBTEST.COM (short username kudu)
I20260929 18:00:33.491710 21105 webserver.cc:460] Webserver started at http://127.7.118.60:40477/ using document root <none> and password file <none>
I20260929 18:00:33.492337 21105 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:33.492439 21105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:00:33.492688 21105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:00:33.495684 21105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-2/data/instance:
uuid: "f7fb12f818584f7f824464571aaf6c2d"
format_stamp: "Formatted at 2026-09-29 18:00:33 on dist-test-slave-r7ph"
server_key: "583253bb54625da50401541e558e5cb5"
server_key_iv: "ab34c556b81c69be387001427a351001"
server_key_version: "encryptionkey@0"
I20260929 18:00:33.496572 21105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-2/wal/instance:
uuid: "f7fb12f818584f7f824464571aaf6c2d"
format_stamp: "Formatted at 2026-09-29 18:00:33 on dist-test-slave-r7ph"
server_key: "583253bb54625da50401541e558e5cb5"
server_key_iv: "ab34c556b81c69be387001427a351001"
server_key_version: "encryptionkey@0"
I20260929 18:00:33.501806 21105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 18:00:33.504657 21121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:33.506304 21105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:00:33.506480 21105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-2/wal
uuid: "f7fb12f818584f7f824464571aaf6c2d"
format_stamp: "Formatted at 2026-09-29 18:00:33 on dist-test-slave-r7ph"
server_key: "583253bb54625da50401541e558e5cb5"
server_key_iv: "ab34c556b81c69be387001427a351001"
server_key_version: "encryptionkey@0"
I20260929 18:00:33.506757 21105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:33.537067 21105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:33.539462 21024 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } attempt: 3
I20260929 18:00:33.540432 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.540044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42137 (local address 127.0.0.1:49194)
0929 18:00:33.540249 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.540321 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 18:00:33.541329 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.541042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42137 (local address 127.0.0.1:49208)
0929 18:00:33.541182 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.541246 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 18:00:33.541509 21024 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.60:42137: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42137: connect: Connection refused (error 111)
I20260929 18:00:33.541841 21105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:33.542091 21105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:33.550172 21105 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.60:42137
I20260929 18:00:33.550191 21173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.60:42137 every 8 connection(s)
I20260929 18:00:33.551297 21105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-2/data/info.pb
I20260929 18:00:33.554025 21099 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } attempt: 3
I20260929 18:00:33.554598 21174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:00:33.555223 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 21105
I20260929 18:00:33.555346 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704657566865-7640-0/minicluster-data/master-2/wal/instance
I20260929 18:00:33.555667 7640 external_mini_cluster.cc:1428] Setting key 721879917e48778f2e2b7e347fa4769f
I20260929 18:00:33.560985 21174 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45759 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:00:33 dist-test-slave-r7ph krb5kdc[20943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704833, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.61@KRBTEST.COM for kudu/127.7.118.60@KRBTEST.COM
Sep 29 18:00:33 dist-test-slave-r7ph krb5kdc[20943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.60@KRBTEST.COM
Sep 29 18:00:33 dist-test-slave-r7ph krb5kdc[20943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704833, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.60@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:00:33.571244 21103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.554727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42137 (local address 127.0.0.1:49222)
0929 18:00:33.554941 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.554957 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:33.555055 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:33.558032 (+ 2977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:33.558036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:33.558301 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:00:33.558426 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:33.558434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.560071 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.560078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:33.560915 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:33.560928 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.562112 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.562116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:33.562208 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:33.563043 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:00:33.563061 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:00:33.565321 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:00:33.568782 (+ 3461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:33.568786 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:33.568789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:33.569154 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:33.569514 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:33.569517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:33.569519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:33.569609 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:33.570648 (+ 1039us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:00:33.570654 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:00:33.570776 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 18:00:33.570869 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 18:00:33.571068 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260929 18:00:33.573109 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.555729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49222 (local address 127.7.118.60:42137)
0929 18:00:33.556296 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:33.556324 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:33.556364 (+ 40us) server_negotiation.cc:407] Connection header received
0929 18:00:33.557293 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:33.557315 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:33.557712 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:33.558080 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:33.558809 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.559928 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.561072 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.561951 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.562127 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:33.565522 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:33.565528 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:33.565541 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:33.565575 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:33.568623 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.569290 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.569295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.569296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.569397 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.569706 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.569708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.569709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.570497 (+ 788us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:33.571580 (+ 1083us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:33.571656 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 18:00:33.571978 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":281,"server-negotiator.queue_time_us":402,"spinlock_wait_cycles":17152,"thread_start_us":231,"threads_started":1}
I20260929 18:00:33.573134 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.556796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49230 (local address 127.7.118.60:42137)
0929 18:00:33.557068 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:33.557075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:33.557087 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:00:33.557334 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:33.557339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:33.557924 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:33.558080 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:33.559386 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.560186 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.561317 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.561945 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.562140 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:33.565594 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:33.565598 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:33.565603 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:33.565631 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:33.568629 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.569193 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.569203 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.569213 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.569404 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.569778 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.569784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.569787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.570503 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:33.570841 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:33.571157 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 18:00:33.571924 (+ 767us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1232,"server-negotiator.queue_time_us":213,"spinlock_wait_cycles":110336,"thread_start_us":79,"threads_started":1}
I20260929 18:00:33.574923 21179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.563572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45759 (local address 127.0.0.1:51676)
0929 18:00:33.564062 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.564119 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:33.564248 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:33.564576 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:33.564582 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:33.565060 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:00:33.565209 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:33.565215 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.566327 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.566334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:33.567438 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:33.567450 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.567987 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.567991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:33.568124 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:33.568667 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:00:33.568679 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:00:33.571039 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:00:33.573011 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:33.573023 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:33.573029 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:33.573373 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:33.573738 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:33.573740 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:33.573741 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:33.573814 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:33.574345 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:00:33.574354 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:00:33.574504 (+ 150us) client_negotiation.cc:769] Sending connection context
0929 18:00:33.574634 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 18:00:33.574793 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":153,"threads_started":1}
I20260929 18:00:33.575310 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.563616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51676 (local address 127.7.118.62:45759)
0929 18:00:33.563759 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:33.563764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:33.564223 (+ 459us) server_negotiation.cc:407] Connection header received
0929 18:00:33.564336 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:33.564339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:33.564396 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:33.564494 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:33.565364 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.566156 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.567582 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.567857 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.567980 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:33.571235 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:33.571240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:33.571243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:33.571278 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:33.572881 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.573504 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.573508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.573510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.573617 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.573914 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.573917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.573919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.574162 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:33.574321 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:33.574725 (+ 404us) server_negotiation.cc:299] Negotiation successful
0929 18:00:33.574856 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:00:33.576439 21174 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.61" port: 39499 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:00:33.578656 21099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906: Bootstrap starting.
I20260929 18:00:33.581482 21099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906: Neither blocks nor log segments found. Creating new log.
I20260929 18:00:33.582412 21099 log.cc:824] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:00:33 dist-test-slave-r7ph krb5kdc[20943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704833, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.60@KRBTEST.COM for kudu/127.7.118.61@KRBTEST.COM
Sep 29 18:00:33 dist-test-slave-r7ph krb5kdc[20943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:00:33.585450 21099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906: No bootstrap required, opened a new log
I20260929 18:00:33.588248 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.577140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53438 (local address 127.7.118.61:39499)
0929 18:00:33.577261 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:33.577265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:33.577280 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:00:33.577321 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:33.577324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:33.577367 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:33.577451 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:33.577945 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.578723 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.579844 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.580178 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.580285 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:33.583350 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:33.583354 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:33.583356 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:33.583382 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:33.585361 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.586751 (+ 1390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.586755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.586757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.586856 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.587208 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.587215 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.587219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.587444 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:33.587561 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:33.587905 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 18:00:33.588058 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:00:33.588519 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.576659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51680 (local address 127.7.118.62:45759)
0929 18:00:33.576772 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:33.576777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:33.576796 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:00:33.576843 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:33.576847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:33.576905 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:33.576985 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:33.577991 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.578642 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.579853 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.580092 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.580161 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:33.584325 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:33.584340 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:33.584346 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:33.584448 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:33.586182 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.586832 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.586836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.586838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.586933 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.587283 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.587286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.587288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.587526 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:33.587879 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:33.588027 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 18:00:33.588166 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:00:33.588729 21179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.576998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:39499 (local address 127.0.0.1:53438)
0929 18:00:33.577155 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.577169 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:33.577245 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:33.577495 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:33.577499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:33.577697 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:00:33.577815 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:33.577822 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.578853 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.578857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:33.579704 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:33.579714 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.580319 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.580323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:33.580431 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:33.581244 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:00:33.581280 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:00:33.583154 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:00:33.585537 (+ 2383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:33.585562 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:33.585568 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:33.586634 (+ 1066us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:33.586971 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:33.586974 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:33.586976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:33.587090 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:33.587561 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:00:33.587565 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:00:33.587701 (+ 136us) client_negotiation.cc:769] Sending connection context
0929 18:00:33.588393 (+ 692us) client_negotiation.cc:240] Negotiation successful
0929 18:00:33.588552 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260929 18:00:33.589529 21174 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:00:33.591414 21099 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ee513781524ed6a3a95773165d7402" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45759 } } peers { permanent_uuid: "d0e4d8aafab44e028689c0cbfc49f906" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 39499 } } peers { permanent_uuid: "f7fb12f818584f7f824464571aaf6c2d" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } }
I20260929 18:00:33.591713 21099 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:00:33.591786 21099 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e4d8aafab44e028689c0cbfc49f906, State: Initialized, Role: FOLLOWER
I20260929 18:00:33.592375 21099 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ee513781524ed6a3a95773165d7402" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45759 } } peers { permanent_uuid: "d0e4d8aafab44e028689c0cbfc49f906" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 39499 } } peers { permanent_uuid: "f7fb12f818584f7f824464571aaf6c2d" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } }
I20260929 18:00:33.592935 21181 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ee513781524ed6a3a95773165d7402" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45759 } } peers { permanent_uuid: "d0e4d8aafab44e028689c0cbfc49f906" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 39499 } } peers { permanent_uuid: "f7fb12f818584f7f824464571aaf6c2d" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } } }
I20260929 18:00:33.593189 21181 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:00:33.594532 21099 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 18:00:33 dist-test-slave-r7ph krb5kdc[20943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704833, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.60@KRBTEST.COM for kudu/127.7.118.60@KRBTEST.COM
Sep 29 18:00:33 dist-test-slave-r7ph krb5kdc[20943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.61@KRBTEST.COM
I20260929 18:00:33.600700 21179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.590080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42137 (local address 127.0.0.1:49242)
0929 18:00:33.590371 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.590388 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:33.590486 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:33.590746 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:33.590748 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:33.590939 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:00:33.591011 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:33.591017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.591969 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.591973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:33.592764 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:33.592778 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.593861 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.593865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:33.593945 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:33.594636 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:00:33.594651 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:00:33.596318 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:00:33.598912 (+ 2594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:33.598917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:33.598919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:33.599198 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:33.599512 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:33.599518 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:33.599520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:33.599618 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:33.600140 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:00:33.600142 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:00:33.600245 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 18:00:33.600347 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 18:00:33.600509 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
W20260929 18:00:33.600660 21192 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:00:33.601056 21192 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:00:33.601224 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.590174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53446 (local address 127.7.118.61:39499)
0929 18:00:33.590317 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:33.590322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:33.590337 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:00:33.590439 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:33.590444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:33.590509 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:33.590592 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:33.591500 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.592228 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.593279 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.593556 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.593619 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:33.597149 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:33.597153 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:33.597156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:33.597185 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:33.598812 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.599364 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.599369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.599371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.599459 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.599771 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.599775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.599776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.599966 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:33.600177 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:33.600929 (+ 752us) server_negotiation.cc:299] Negotiation successful
0929 18:00:33.601071 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:00:33.601279 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.590235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49242 (local address 127.7.118.60:42137)
0929 18:00:33.590372 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:33.590376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:33.590433 (+ 57us) server_negotiation.cc:407] Connection header received
0929 18:00:33.590591 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:33.590594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:33.590635 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:33.590737 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:33.591130 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.591863 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.592895 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.593495 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.593651 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:33.596497 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:33.596501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:33.596504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:33.596544 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:33.598760 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.599315 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.599320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.599322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.599400 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.599712 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.599718 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.599723 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.600017 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:33.600104 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:33.601016 (+ 912us) server_negotiation.cc:299] Negotiation successful
0929 18:00:33.601122 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 18:00:33.599658 21031 master_runner.cc:186] Error getting master registration for 127.7.118.62:45759: OK, instance_id { permanent_uuid: "15ee513781524ed6a3a95773165d7402" instance_seqno: 1790704833058132 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:00:33.603763 21031 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:00:33.604593 21174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7fb12f818584f7f824464571aaf6c2d: Bootstrap starting.
I20260929 18:00:33.607571 21174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7fb12f818584f7f824464571aaf6c2d: Neither blocks nor log segments found. Creating new log.
I20260929 18:00:33.607837 7640 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:00:33.608582 21174 log.cc:824] T 00000000000000000000000000000000 P f7fb12f818584f7f824464571aaf6c2d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:00:33.612254 21174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7fb12f818584f7f824464571aaf6c2d: No bootstrap required, opened a new log
I20260929 18:00:33.617115 21174 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7fb12f818584f7f824464571aaf6c2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ee513781524ed6a3a95773165d7402" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45759 } } peers { permanent_uuid: "d0e4d8aafab44e028689c0cbfc49f906" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 39499 } } peers { permanent_uuid: "f7fb12f818584f7f824464571aaf6c2d" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } }
I20260929 18:00:33.617477 21174 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7fb12f818584f7f824464571aaf6c2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:00:33.617540 21174 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7fb12f818584f7f824464571aaf6c2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7fb12f818584f7f824464571aaf6c2d, State: Initialized, Role: FOLLOWER
I20260929 18:00:33.618152 21174 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7fb12f818584f7f824464571aaf6c2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ee513781524ed6a3a95773165d7402" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45759 } } peers { permanent_uuid: "d0e4d8aafab44e028689c0cbfc49f906" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 39499 } } peers { permanent_uuid: "f7fb12f818584f7f824464571aaf6c2d" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } }
I20260929 18:00:33.620021 21174 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7fb12f818584f7f824464571aaf6c2d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:00:33.620922 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.610399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51682 (local address 127.7.118.62:45759)
0929 18:00:33.610892 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:33.610896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:33.610914 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:00:33.610978 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:33.610981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:33.611032 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:33.611121 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:33.612162 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.612916 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.614145 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.614507 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.614600 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:33.616732 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:33.616736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:33.616738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:33.616774 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:33.618632 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.619235 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.619244 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.619246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.619378 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.619760 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.619763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.619765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.620024 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:33.620155 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:33.620574 (+ 419us) server_negotiation.cc:299] Negotiation successful
0929 18:00:33.620723 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407}
I20260929 18:00:33.627244 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.610891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49256 (local address 127.7.118.60:42137)
0929 18:00:33.611021 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:33.611026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:33.615822 (+ 4796us) server_negotiation.cc:407] Connection header received
0929 18:00:33.616035 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:33.616039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:33.616092 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:33.616148 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:33.617430 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.618689 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.619993 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.620339 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.620421 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:33.622741 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:33.622746 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:33.622750 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:33.622782 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:33.624889 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.625416 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.625419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.625421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.625517 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.625867 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.625870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.625872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.626088 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:33.626243 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:33.626923 (+ 680us) server_negotiation.cc:299] Negotiation successful
0929 18:00:33.627056 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:00:33.625240 21105 master_runner.cc:186] Error getting master registration for 127.7.118.62:45759: OK, instance_id { permanent_uuid: "15ee513781524ed6a3a95773165d7402" instance_seqno: 1790704833058132 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:00:33.629318 21105 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:00:33.630039 21201 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7fb12f818584f7f824464571aaf6c2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ee513781524ed6a3a95773165d7402" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45759 } } peers { permanent_uuid: "d0e4d8aafab44e028689c0cbfc49f906" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 39499 } } peers { permanent_uuid: "f7fb12f818584f7f824464571aaf6c2d" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } } }
I20260929 18:00:33.630203 21201 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7fb12f818584f7f824464571aaf6c2d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:00:33.632860 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.622609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53462 (local address 127.7.118.61:39499)
0929 18:00:33.622740 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:33.622744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:33.623125 (+ 381us) server_negotiation.cc:407] Connection header received
0929 18:00:33.623236 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:33.623240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:33.623293 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:33.623373 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:33.624469 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.625289 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.626464 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.626822 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.626913 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:33.629405 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:33.629410 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:33.629413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:33.629452 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:33.631066 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.631551 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.631555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.631557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.631653 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.631944 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.631948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.631949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.632157 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:33.632296 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:33.632551 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 18:00:33.632681 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:00:33.633410 21181 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:00:33.633567 21181 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ee513781524ed6a3a95773165d7402" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45759 } } peers { permanent_uuid: "d0e4d8aafab44e028689c0cbfc49f906" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 39499 } } peers { permanent_uuid: "f7fb12f818584f7f824464571aaf6c2d" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } }
W20260929 18:00:33.634366 21212 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7fb12f818584f7f824464571aaf6c2d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:00:33.634451 21212 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:00:33.634821 21181 leader_election.cc:288] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ee513781524ed6a3a95773165d7402 (127.7.118.62:45759), f7fb12f818584f7f824464571aaf6c2d (127.7.118.60:42137)
I20260929 18:00:33.635397 21148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0e4d8aafab44e028689c0cbfc49f906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7fb12f818584f7f824464571aaf6c2d" is_pre_election: true
I20260929 18:00:33.635752 21148 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f7fb12f818584f7f824464571aaf6c2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0e4d8aafab44e028689c0cbfc49f906 in term 0.
I20260929 18:00:33.636274 21048 leader_election.cc:302] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0e4d8aafab44e028689c0cbfc49f906, f7fb12f818584f7f824464571aaf6c2d; no voters:
I20260929 18:00:33.636581 21181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:00:33.636679 21181 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:00:33.636724 21181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:00:33.638238 21181 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ee513781524ed6a3a95773165d7402" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45759 } } peers { permanent_uuid: "d0e4d8aafab44e028689c0cbfc49f906" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 39499 } } peers { permanent_uuid: "f7fb12f818584f7f824464571aaf6c2d" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } }
I20260929 18:00:33.638960 21181 leader_election.cc:288] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [CANDIDATE]: Term 1 election: Requested vote from peers 15ee513781524ed6a3a95773165d7402 (127.7.118.62:45759), f7fb12f818584f7f824464571aaf6c2d (127.7.118.60:42137)
I20260929 18:00:33.639822 21148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0e4d8aafab44e028689c0cbfc49f906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7fb12f818584f7f824464571aaf6c2d"
I20260929 18:00:33.640004 21148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7fb12f818584f7f824464571aaf6c2d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:00:33.641618 21148 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f7fb12f818584f7f824464571aaf6c2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e4d8aafab44e028689c0cbfc49f906 in term 1.
I20260929 18:00:33.642853 21048 leader_election.cc:302] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0e4d8aafab44e028689c0cbfc49f906, f7fb12f818584f7f824464571aaf6c2d; no voters:
I20260929 18:00:33.643216 21181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:00:33.643849 21181 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [term 1 LEADER]: Becoming Leader. State: Replica: d0e4d8aafab44e028689c0cbfc49f906, State: Running, Role: LEADER
I20260929 18:00:33.644472 21181 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ee513781524ed6a3a95773165d7402" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45759 } } peers { permanent_uuid: "d0e4d8aafab44e028689c0cbfc49f906" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 39499 } } peers { permanent_uuid: "f7fb12f818584f7f824464571aaf6c2d" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } }
I20260929 18:00:33.646545 21181 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0e4d8aafab44e028689c0cbfc49f906. Latest consensus state: current_term: 1 leader_uuid: "d0e4d8aafab44e028689c0cbfc49f906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ee513781524ed6a3a95773165d7402" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45759 } } peers { permanent_uuid: "d0e4d8aafab44e028689c0cbfc49f906" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 39499 } } peers { permanent_uuid: "f7fb12f818584f7f824464571aaf6c2d" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } } }
I20260929 18:00:33.646687 21181 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [sys.catalog]: This master's current role is: LEADER
I20260929 18:00:33.658674 21216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:00:33.659760 21216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:00:33.668458 21148 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7fb12f818584f7f824464571aaf6c2d [term 1 FOLLOWER]: Refusing update from remote peer d0e4d8aafab44e028689c0cbfc49f906: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:00:33.669309 21214 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7fb12f818584f7f824464571aaf6c2d" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:00:33.674469 21201 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7fb12f818584f7f824464571aaf6c2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0e4d8aafab44e028689c0cbfc49f906. Latest consensus state: current_term: 1 leader_uuid: "d0e4d8aafab44e028689c0cbfc49f906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ee513781524ed6a3a95773165d7402" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45759 } } peers { permanent_uuid: "d0e4d8aafab44e028689c0cbfc49f906" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 39499 } } peers { permanent_uuid: "f7fb12f818584f7f824464571aaf6c2d" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } } }
I20260929 18:00:33.674671 21201 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7fb12f818584f7f824464571aaf6c2d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:00:33.681660 21214 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0e4d8aafab44e028689c0cbfc49f906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ee513781524ed6a3a95773165d7402" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45759 } } peers { permanent_uuid: "d0e4d8aafab44e028689c0cbfc49f906" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 39499 } } peers { permanent_uuid: "f7fb12f818584f7f824464571aaf6c2d" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } } }
I20260929 18:00:33.681838 21214 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [sys.catalog]: This master's current role is: LEADER
I20260929 18:00:33.682999 21201 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7fb12f818584f7f824464571aaf6c2d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0e4d8aafab44e028689c0cbfc49f906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ee513781524ed6a3a95773165d7402" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45759 } } peers { permanent_uuid: "d0e4d8aafab44e028689c0cbfc49f906" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 39499 } } peers { permanent_uuid: "f7fb12f818584f7f824464571aaf6c2d" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } } }
I20260929 18:00:33.683164 21201 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7fb12f818584f7f824464571aaf6c2d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:00:33.687719 21216 catalog_manager.cc:1347] Generated new cluster ID: 772c1f5268f343128c15ebad41c3c51e
I20260929 18:00:33.687810 21216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:00:33.705688 21024 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } attempt: 4
Sep 29 18:00:33 dist-test-slave-r7ph krb5kdc[20943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704833, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for kudu/127.7.118.60@KRBTEST.COM
I20260929 18:00:33.722723 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.706784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42137 (local address 127.0.0.1:49258)
0929 18:00:33.706970 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:33.706983 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:33.707079 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:33.707425 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:33.707428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:33.707646 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:00:33.707746 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:33.707752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.709226 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.709230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:33.710198 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:33.710211 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.711368 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.711372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:33.711477 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:33.714427 (+ 2950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:00:33.714444 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:00:33.716751 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:00:33.720659 (+ 3908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:33.720663 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:33.720680 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:33.721006 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:33.721426 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:33.721429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:33.721432 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:33.721543 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:33.722074 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:00:33.722077 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:00:33.722190 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 18:00:33.722325 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 18:00:33.722501 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260929 18:00:33.723232 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:33.706513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49258 (local address 127.7.118.60:42137)
0929 18:00:33.706666 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:33.706671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:33.707155 (+ 484us) server_negotiation.cc:407] Connection header received
0929 18:00:33.707204 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:33.707206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:33.707250 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:33.707315 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:33.707876 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.708948 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.710536 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:33.711010 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:33.711128 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:33.717695 (+ 6567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:33.717699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:33.717703 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:33.717734 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:33.720461 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.721174 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.721178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.721180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.721286 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:33.721661 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:33.721665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:33.721667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:33.721928 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:33.722022 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:33.722959 (+ 937us) server_negotiation.cc:299] Negotiation successful
0929 18:00:33.723082 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:00:33.723928 21216 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:00:33.726883 21024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15ee513781524ed6a3a95773165d7402: Bootstrap starting.
I20260929 18:00:33.729627 21216 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:00:33.730732 21024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15ee513781524ed6a3a95773165d7402: Neither blocks nor log segments found. Creating new log.
I20260929 18:00:33.731756 21024 log.cc:824] T 00000000000000000000000000000000 P 15ee513781524ed6a3a95773165d7402: Log is configured to *not* fsync() on all Append() calls
I20260929 18:00:33.747092 21024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15ee513781524ed6a3a95773165d7402: No bootstrap required, opened a new log
I20260929 18:00:33.752923 21216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906: Generated new TSK 0
I20260929 18:00:33.754765 21216 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:00:33.758188 21024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15ee513781524ed6a3a95773165d7402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ee513781524ed6a3a95773165d7402" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45759 } } peers { permanent_uuid: "d0e4d8aafab44e028689c0cbfc49f906" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 39499 } } peers { permanent_uuid: "f7fb12f818584f7f824464571aaf6c2d" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } }
I20260929 18:00:33.758538 21024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15ee513781524ed6a3a95773165d7402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:00:33.758601 21024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15ee513781524ed6a3a95773165d7402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ee513781524ed6a3a95773165d7402, State: Initialized, Role: FOLLOWER
I20260929 18:00:33.759223 21024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15ee513781524ed6a3a95773165d7402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ee513781524ed6a3a95773165d7402" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45759 } } peers { permanent_uuid: "d0e4d8aafab44e028689c0cbfc49f906" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 39499 } } peers { permanent_uuid: "f7fb12f818584f7f824464571aaf6c2d" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } }
I20260929 18:00:33.761309 21024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15ee513781524ed6a3a95773165d7402 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:00:33.765688 20998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0e4d8aafab44e028689c0cbfc49f906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ee513781524ed6a3a95773165d7402" is_pre_election: true
I20260929 18:00:33.766311 20997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0e4d8aafab44e028689c0cbfc49f906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ee513781524ed6a3a95773165d7402"
I20260929 18:00:33.766193 20996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15ee513781524ed6a3a95773165d7402 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:00:33.769174 20996 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 15ee513781524ed6a3a95773165d7402 [term 1 FOLLOWER]: Refusing update from remote peer d0e4d8aafab44e028689c0cbfc49f906: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:00:33.771385 21214 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ee513781524ed6a3a95773165d7402" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 18:00:33.774878 21234 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 15ee513781524ed6a3a95773165d7402: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:00:33.774999 21234 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:00:33.783532 21223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15ee513781524ed6a3a95773165d7402 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0e4d8aafab44e028689c0cbfc49f906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ee513781524ed6a3a95773165d7402" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45759 } } peers { permanent_uuid: "d0e4d8aafab44e028689c0cbfc49f906" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 39499 } } peers { permanent_uuid: "f7fb12f818584f7f824464571aaf6c2d" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } } }
I20260929 18:00:33.783738 21223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15ee513781524ed6a3a95773165d7402 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:00:33.783859 21223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15ee513781524ed6a3a95773165d7402 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0e4d8aafab44e028689c0cbfc49f906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ee513781524ed6a3a95773165d7402" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45759 } } peers { permanent_uuid: "d0e4d8aafab44e028689c0cbfc49f906" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 39499 } } peers { permanent_uuid: "f7fb12f818584f7f824464571aaf6c2d" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } } }
I20260929 18:00:33.783962 21223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15ee513781524ed6a3a95773165d7402 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:00:33.790536 21236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15ee513781524ed6a3a95773165d7402 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0e4d8aafab44e028689c0cbfc49f906. Latest consensus state: current_term: 1 leader_uuid: "d0e4d8aafab44e028689c0cbfc49f906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ee513781524ed6a3a95773165d7402" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45759 } } peers { permanent_uuid: "d0e4d8aafab44e028689c0cbfc49f906" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 39499 } } peers { permanent_uuid: "f7fb12f818584f7f824464571aaf6c2d" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } } }
I20260929 18:00:33.790845 21236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15ee513781524ed6a3a95773165d7402 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:00:33.791468 21237 mvcc.cc:205] Tried to move back new op lower bound from 7334726999037550592 to 7334726998611402752. Current Snapshot: MvccSnapshot[applied={T|T < 7334726998697017344}]
I20260929 18:00:34.386067 21214 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0e4d8aafab44e028689c0cbfc49f906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ee513781524ed6a3a95773165d7402" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45759 } } peers { permanent_uuid: "d0e4d8aafab44e028689c0cbfc49f906" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 39499 } } peers { permanent_uuid: "f7fb12f818584f7f824464571aaf6c2d" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42137 } } }
I20260929 18:00:34.386313 21214 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e4d8aafab44e028689c0cbfc49f906 [sys.catalog]: This master's current role is: LEADER
I20260929 18:00:34.635854 21212 catalog_manager.cc:1259] Loaded cluster ID: 772c1f5268f343128c15ebad41c3c51e
I20260929 18:00:34.635939 21212 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f7fb12f818584f7f824464571aaf6c2d: loading cluster ID for follower catalog manager: success
I20260929 18:00:34.637825 21212 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f7fb12f818584f7f824464571aaf6c2d: acquiring CA information for follower catalog manager: success
I20260929 18:00:34.639256 21212 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f7fb12f818584f7f824464571aaf6c2d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:00:34.776571 21234 catalog_manager.cc:1259] Loaded cluster ID: 772c1f5268f343128c15ebad41c3c51e
I20260929 18:00:34.776746 21234 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 15ee513781524ed6a3a95773165d7402: loading cluster ID for follower catalog manager: success
I20260929 18:00:34.778867 21234 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 15ee513781524ed6a3a95773165d7402: acquiring CA information for follower catalog manager: success
I20260929 18:00:34.780341 21234 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 15ee513781524ed6a3a95773165d7402: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:00:34.851213 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 20956
I20260929 18:00:34.858450 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 21031
I20260929 18:00:34.864908 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 21105
2026-09-29T18:00:34Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3191 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:00:34 dist-test-slave-r7ph krb5kdc[21252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:00:34 dist-test-slave-r7ph krb5kdc[21252](info): set up 2 sockets
Sep 29 18:00:34 dist-test-slave-r7ph krb5kdc[21252](info): commencing operation
krb5kdc: starting...
W20260929 18:00:35.958570 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 18:00:36 dist-test-slave-r7ph krb5kdc[21252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:00:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:00:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 18:00:36.102393 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:35737
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:41221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:35737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:00:36.225265 21265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:36.225682 21265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:36.225775 21265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:00:36.230131 21265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:00:36.230258 21265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:36.230338 21265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:36.230402 21265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:00:36.230443 21265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:00:36.236361 21265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:41221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:35737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:35737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:36.237637 21265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:36.238854 21265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:00:36.248605 21273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:36.248673 21265 server_base.cc:1034] running on GCE node
W20260929 18:00:36.248608 21270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:36.248605 21271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:36.249622 21265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:36.250999 21265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:36.252228 21265 hybrid_clock.cc:648] HybridClock initialized: now 1790704836252186 us; error 60 us; skew 500 ppm
Sep 29 18:00:36 dist-test-slave-r7ph krb5kdc[21252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704836, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:36.255585 21265 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 18:00:36.256611 21265 webserver.cc:460] Webserver started at http://127.7.118.62:36739/ using document root <none> and password file <none>
I20260929 18:00:36.257212 21265 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:36.257279 21265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:00:36.257457 21265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:00:36.259784 21265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "d893d874b1dd474897828e297aec9fcd"
format_stamp: "Formatted at 2026-09-29 18:00:36 on dist-test-slave-r7ph"
server_key: "09a82c84642d87c7b3866ec2c3bde6cb"
server_key_iv: "4f198c427ce22fbc5f2954342e9d1a1e"
server_key_version: "encryptionkey@0"
I20260929 18:00:36.260375 21265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "d893d874b1dd474897828e297aec9fcd"
format_stamp: "Formatted at 2026-09-29 18:00:36 on dist-test-slave-r7ph"
server_key: "09a82c84642d87c7b3866ec2c3bde6cb"
server_key_iv: "4f198c427ce22fbc5f2954342e9d1a1e"
server_key_version: "encryptionkey@0"
I20260929 18:00:36.264564 21265 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:00:36.267489 21280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:36.269088 21265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:00:36.269236 21265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "d893d874b1dd474897828e297aec9fcd"
format_stamp: "Formatted at 2026-09-29 18:00:36 on dist-test-slave-r7ph"
server_key: "09a82c84642d87c7b3866ec2c3bde6cb"
server_key_iv: "4f198c427ce22fbc5f2954342e9d1a1e"
server_key_version: "encryptionkey@0"
I20260929 18:00:36.269353 21265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:36.283825 21265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:36.287740 21265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:36.287933 21265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:36.295773 21265 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:35737
I20260929 18:00:36.295791 21332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:35737 every 8 connection(s)
I20260929 18:00:36.296888 21265 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:00:36.299571 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 21265
I20260929 18:00:36.299678 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:00:36.300005 7640 external_mini_cluster.cc:1428] Setting key 238206ae4e07aded99ac44e8e997cce1
I20260929 18:00:36.300374 21333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:00:36.307849 21333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d893d874b1dd474897828e297aec9fcd: Bootstrap starting.
Sep 29 18:00:36 dist-test-slave-r7ph krb5kdc[21252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:00:36.310684 21333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d893d874b1dd474897828e297aec9fcd: Neither blocks nor log segments found. Creating new log.
I20260929 18:00:36.311656 21333 log.cc:824] T 00000000000000000000000000000000 P d893d874b1dd474897828e297aec9fcd: Log is configured to *not* fsync() on all Append() calls
I20260929 18:00:36.314350 21333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d893d874b1dd474897828e297aec9fcd: No bootstrap required, opened a new log
I20260929 18:00:36.315549 21336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:36.301602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35158 (local address 127.7.118.62:35737)
0929 18:00:36.302145 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:36.302162 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:36.302202 (+ 40us) server_negotiation.cc:407] Connection header received
0929 18:00:36.302995 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:36.303014 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:36.303383 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:36.303755 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:36.304616 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:36.305667 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:36.306636 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:36.307180 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:36.307364 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:36.310166 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:36.310171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:36.310187 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:36.310223 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:36.312305 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:36.312871 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:36.312877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:36.312882 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:36.313045 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:36.313367 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:36.313374 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:36.313377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:36.313808 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:36.314040 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:36.314402 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 18:00:36.314766 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":182,"threads_started":1}
I20260929 18:00:36.318292 21333 raft_consensus.cc:348] T 00000000000000000000000000000000 P d893d874b1dd474897828e297aec9fcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d893d874b1dd474897828e297aec9fcd" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35737 } }
I20260929 18:00:36.318673 21333 raft_consensus.cc:374] T 00000000000000000000000000000000 P d893d874b1dd474897828e297aec9fcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:00:36.318768 21333 raft_consensus.cc:729] T 00000000000000000000000000000000 P d893d874b1dd474897828e297aec9fcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d893d874b1dd474897828e297aec9fcd, State: Initialized, Role: FOLLOWER
I20260929 18:00:36.319346 21333 consensus_queue.cc:260] T 00000000000000000000000000000000 P d893d874b1dd474897828e297aec9fcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d893d874b1dd474897828e297aec9fcd" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35737 } }
I20260929 18:00:36.319533 21333 raft_consensus.cc:388] T 00000000000000000000000000000000 P d893d874b1dd474897828e297aec9fcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:00:36.319615 21333 raft_consensus.cc:482] T 00000000000000000000000000000000 P d893d874b1dd474897828e297aec9fcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:00:36.319621 7640 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:00:36.319715 21333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d893d874b1dd474897828e297aec9fcd [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:00:36.320938 21333 raft_consensus.cc:504] T 00000000000000000000000000000000 P d893d874b1dd474897828e297aec9fcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d893d874b1dd474897828e297aec9fcd" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35737 } }
I20260929 18:00:36.321364 21333 leader_election.cc:302] T 00000000000000000000000000000000 P d893d874b1dd474897828e297aec9fcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d893d874b1dd474897828e297aec9fcd; no voters:
I20260929 18:00:36.321719 21333 leader_election.cc:288] T 00000000000000000000000000000000 P d893d874b1dd474897828e297aec9fcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:00:36.321864 21338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d893d874b1dd474897828e297aec9fcd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:00:36.322098 21338 raft_consensus.cc:686] T 00000000000000000000000000000000 P d893d874b1dd474897828e297aec9fcd [term 1 LEADER]: Becoming Leader. State: Replica: d893d874b1dd474897828e297aec9fcd, State: Running, Role: LEADER
I20260929 18:00:36.322486 21338 consensus_queue.cc:237] T 00000000000000000000000000000000 P d893d874b1dd474897828e297aec9fcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d893d874b1dd474897828e297aec9fcd" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35737 } }
I20260929 18:00:36.323064 21333 sys_catalog.cc:564] T 00000000000000000000000000000000 P d893d874b1dd474897828e297aec9fcd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:00:36.324312 21340 sys_catalog.cc:455] T 00000000000000000000000000000000 P d893d874b1dd474897828e297aec9fcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d893d874b1dd474897828e297aec9fcd. Latest consensus state: current_term: 1 leader_uuid: "d893d874b1dd474897828e297aec9fcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d893d874b1dd474897828e297aec9fcd" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35737 } } }
I20260929 18:00:36.324471 21340 sys_catalog.cc:458] T 00000000000000000000000000000000 P d893d874b1dd474897828e297aec9fcd [sys.catalog]: This master's current role is: LEADER
I20260929 18:00:36.324820 21339 sys_catalog.cc:455] T 00000000000000000000000000000000 P d893d874b1dd474897828e297aec9fcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d893d874b1dd474897828e297aec9fcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d893d874b1dd474897828e297aec9fcd" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35737 } } }
I20260929 18:00:36.324941 21339 sys_catalog.cc:458] T 00000000000000000000000000000000 P d893d874b1dd474897828e297aec9fcd [sys.catalog]: This master's current role is: LEADER
I20260929 18:00:36.324906 21347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:00:36.328641 21347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:00:36.335578 21347 catalog_manager.cc:1347] Generated new cluster ID: dd921432bf1a40ad857da8fd9a44f0f6
I20260929 18:00:36.335675 21347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:00:36.358484 21347 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:00:36.359643 21347 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:00:36.365608 21347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d893d874b1dd474897828e297aec9fcd: Generated new TSK 0
I20260929 18:00:36.366534 21347 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:00:36.375974 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 21265
2026-09-29T18:00:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1509 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:00:36 dist-test-slave-r7ph krb5kdc[21360](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:00:36 dist-test-slave-r7ph krb5kdc[21360](info): set up 2 sockets
Sep 29 18:00:36 dist-test-slave-r7ph krb5kdc[21360](info): commencing operation
krb5kdc: starting...
W20260929 18:00:37.426577 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 18:00:37 dist-test-slave-r7ph krb5kdc[21360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:00:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:00:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 18:00:37.571116 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:33763
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:33763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:00:37.696861 21373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:37.697268 21373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:37.697388 21373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:00:37.702191 21373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:00:37.702351 21373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:37.702395 21373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:37.702438 21373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:00:37.702503 21373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:00:37.708859 21373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:33763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:33763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:37.710176 21373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:37.711400 21373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:00:37.721310 21381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:37.721257 21379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:37.721452 21373 server_base.cc:1034] running on GCE node
W20260929 18:00:37.721257 21378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:37.722455 21373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:37.723713 21373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:37.724942 21373 hybrid_clock.cc:648] HybridClock initialized: now 1790704837724912 us; error 46 us; skew 500 ppm
Sep 29 18:00:37 dist-test-slave-r7ph krb5kdc[21360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704837, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:37.741858 21373 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 18:00:37.743367 21373 webserver.cc:460] Webserver started at http://127.7.118.62:39109/ using document root <none> and password file <none>
I20260929 18:00:37.744415 21373 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:37.744536 21373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:00:37.744827 21373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:00:37.748432 21373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "8a24a1dd183f432da92e1a3f81726627"
format_stamp: "Formatted at 2026-09-29 18:00:37 on dist-test-slave-r7ph"
server_key: "3e5f2ab3a17cbd435f56dd7f3c28236d"
server_key_iv: "9c208ea10dd20b632694d814569798ce"
server_key_version: "encryptionkey@0"
I20260929 18:00:37.749279 21373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "8a24a1dd183f432da92e1a3f81726627"
format_stamp: "Formatted at 2026-09-29 18:00:37 on dist-test-slave-r7ph"
server_key: "3e5f2ab3a17cbd435f56dd7f3c28236d"
server_key_iv: "9c208ea10dd20b632694d814569798ce"
server_key_version: "encryptionkey@0"
I20260929 18:00:37.755156 21373 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 18:00:37.758890 21388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:37.760573 21373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:00:37.760761 21373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "8a24a1dd183f432da92e1a3f81726627"
format_stamp: "Formatted at 2026-09-29 18:00:37 on dist-test-slave-r7ph"
server_key: "3e5f2ab3a17cbd435f56dd7f3c28236d"
server_key_iv: "9c208ea10dd20b632694d814569798ce"
server_key_version: "encryptionkey@0"
I20260929 18:00:37.760941 21373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:37.794494 21373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:37.798667 21373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:37.798920 21373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:37.807466 21373 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:33763
I20260929 18:00:37.807767 21440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:33763 every 8 connection(s)
I20260929 18:00:37.808655 21373 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:00:37.809749 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 21373
I20260929 18:00:37.809911 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:00:37.810292 7640 external_mini_cluster.cc:1428] Setting key 147500998b569769757cf75516020947
I20260929 18:00:37.812249 21441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:00:37.820488 21441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627: Bootstrap starting.
Sep 29 18:00:37 dist-test-slave-r7ph krb5kdc[21360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:00:37.823669 21441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627: Neither blocks nor log segments found. Creating new log.
I20260929 18:00:37.824775 21441 log.cc:824] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627: Log is configured to *not* fsync() on all Append() calls
I20260929 18:00:37.827410 21441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627: No bootstrap required, opened a new log
I20260929 18:00:37.828423 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:37.811907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32888 (local address 127.7.118.62:33763)
0929 18:00:37.812391 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:37.812411 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:37.812459 (+ 48us) server_negotiation.cc:407] Connection header received
0929 18:00:37.813346 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:37.813372 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:37.813850 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:37.814321 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:37.815164 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:37.816806 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:37.817912 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:37.818862 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:37.819241 (+ 379us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:37.822095 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:37.822103 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:37.822123 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:37.822169 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:37.824608 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:37.825288 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:37.825297 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:37.825304 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:37.825456 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:37.825820 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:37.825826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:37.825828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:37.826403 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:37.826643 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:37.827005 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 18:00:37.827339 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":122,"threads_started":1}
I20260929 18:00:37.831166 21441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a24a1dd183f432da92e1a3f81726627" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33763 } }
I20260929 18:00:37.831436 21441 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:00:37.831521 21441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a24a1dd183f432da92e1a3f81726627, State: Initialized, Role: FOLLOWER
I20260929 18:00:37.831974 21441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a24a1dd183f432da92e1a3f81726627" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33763 } }
I20260929 18:00:37.832127 21441 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:00:37.832214 21441 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:00:37.832335 21441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:00:37.833343 21441 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a24a1dd183f432da92e1a3f81726627" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33763 } }
I20260929 18:00:37.833714 21441 leader_election.cc:302] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a24a1dd183f432da92e1a3f81726627; no voters:
I20260929 18:00:37.834108 21441 leader_election.cc:288] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:00:37.834340 21446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:00:37.834627 21446 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [term 1 LEADER]: Becoming Leader. State: Replica: 8a24a1dd183f432da92e1a3f81726627, State: Running, Role: LEADER
I20260929 18:00:37.835005 21446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a24a1dd183f432da92e1a3f81726627" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33763 } }
I20260929 18:00:37.835633 21441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:00:37.837194 21448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a24a1dd183f432da92e1a3f81726627. Latest consensus state: current_term: 1 leader_uuid: "8a24a1dd183f432da92e1a3f81726627" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a24a1dd183f432da92e1a3f81726627" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33763 } } }
I20260929 18:00:37.837435 21448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [sys.catalog]: This master's current role is: LEADER
I20260929 18:00:37.837189 21447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a24a1dd183f432da92e1a3f81726627" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a24a1dd183f432da92e1a3f81726627" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33763 } } }
I20260929 18:00:37.837507 21447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [sys.catalog]: This master's current role is: LEADER
W20260929 18:00:37.841259 21461 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:00:37.841383 21461 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:00:37.841467 21458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:00:37.842331 21458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:00:37.849431 21458 catalog_manager.cc:1347] Generated new cluster ID: bb83e00edd5f49829d3c0e9d1eada7cc
I20260929 18:00:37.849570 21458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:00:37.874006 21458 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:00:37.875398 21458 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:00:37.880182 21458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627: Generated new TSK 0
I20260929 18:00:37.880982 21458 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.118.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
I20260929 18:00:37.922744 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:0
--local_ip_for_outbound_sockets=127.7.118.1
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:33763
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:00:38.066502 21467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:38.066901 21467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:38.066951 21467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:00:38.071619 21467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:38.071696 21467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:38.071761 21467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 18:00:38.078770 21467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:33763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:38.080111 21467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:38.081295 21467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:00:38.092818 21475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:38.092811 21472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:38.092809 21473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:38.093129 21467 server_base.cc:1034] running on GCE node
I20260929 18:00:38.093779 21467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:38.094596 21467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:38.095820 21467 hybrid_clock.cc:648] HybridClock initialized: now 1790704838095795 us; error 38 us; skew 500 ppm
Sep 29 18:00:38 dist-test-slave-r7ph krb5kdc[21360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704838, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:38.099357 21467 init.cc:377] Logged in from keytab as kudu/127.7.118.1@KRBTEST.COM (short username kudu)
I20260929 18:00:38.100495 21467 webserver.cc:460] Webserver started at http://127.7.118.1:38141/ using document root <none> and password file <none>
I20260929 18:00:38.101151 21467 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:38.101269 21467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:00:38.101528 21467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:00:38.103951 21467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/data/instance:
uuid: "d0d030cb1647419cbf29b0991405c7ed"
format_stamp: "Formatted at 2026-09-29 18:00:38 on dist-test-slave-r7ph"
server_key: "2d221812c87fbc319150cd86b28ad975"
server_key_iv: "3338f886ffdca4aa7f6513fb8a4716fa"
server_key_version: "encryptionkey@0"
I20260929 18:00:38.104665 21467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance:
uuid: "d0d030cb1647419cbf29b0991405c7ed"
format_stamp: "Formatted at 2026-09-29 18:00:38 on dist-test-slave-r7ph"
server_key: "2d221812c87fbc319150cd86b28ad975"
server_key_iv: "3338f886ffdca4aa7f6513fb8a4716fa"
server_key_version: "encryptionkey@0"
I20260929 18:00:38.109138 21467 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 18:00:38.112159 21482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:38.113935 21467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:00:38.114130 21467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "d0d030cb1647419cbf29b0991405c7ed"
format_stamp: "Formatted at 2026-09-29 18:00:38 on dist-test-slave-r7ph"
server_key: "2d221812c87fbc319150cd86b28ad975"
server_key_iv: "3338f886ffdca4aa7f6513fb8a4716fa"
server_key_version: "encryptionkey@0"
I20260929 18:00:38.114353 21467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:38.145068 21467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:38.149775 21467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:38.150061 21467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:38.150890 21467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:00:38.151917 21467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:00:38.152002 21467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:38.152086 21467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:00:38.152140 21467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:38.163928 21467 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:33205
I20260929 18:00:38.164036 21595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:33205 every 8 connection(s)
I20260929 18:00:38.165011 21467 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 18:00:38.172056 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 21467
I20260929 18:00:38.172178 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance
I20260929 18:00:38.172494 7640 external_mini_cluster.cc:1428] Setting key 07083238e255961bbb7ae7ac98a0f35f
Sep 29 18:00:38 dist-test-slave-r7ph krb5kdc[21360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704838, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.118.2@KRBTEST.COM; defaulting to no policy
I20260929 18:00:38.183668 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:38.167323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:40399 (local address 127.7.118.62:33763)
0929 18:00:38.167499 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:38.167504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:38.168727 (+ 1223us) server_negotiation.cc:407] Connection header received
0929 18:00:38.170333 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:38.170339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:38.170410 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:38.170516 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:38.172028 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:38.172736 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:38.174614 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:38.174872 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:38.174994 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:38.178575 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:38.178579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:38.178582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:38.178614 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:38.180812 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:38.181604 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:38.181609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:38.181611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:38.181716 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:38.182161 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:38.182165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:38.182167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:38.182481 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:38.182604 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:38.183311 (+ 707us) server_negotiation.cc:299] Negotiation successful
0929 18:00:38.183480 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 18:00:38.184744 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:38.167850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.1:40399)
0929 18:00:38.168544 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:38.168598 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:38.169987 (+ 1389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:38.170761 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:38.170769 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:38.171209 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:00:38.171821 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:38.171843 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:38.172887 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:38.172891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:38.174464 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:38.174479 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:38.175050 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:38.175054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:38.175249 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:38.175987 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:00:38.176017 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:00:38.178323 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:00:38.181045 (+ 2722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:38.181054 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:38.181078 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:38.181463 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:38.181834 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:38.181838 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:38.181841 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:38.182040 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:38.182623 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:00:38.182633 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:00:38.183009 (+ 376us) client_negotiation.cc:769] Sending connection context
0929 18:00:38.183245 (+ 236us) client_negotiation.cc:240] Negotiation successful
0929 18:00:38.183685 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":220,"threads_started":1}
I20260929 18:00:38.186494 21596 heartbeater.cc:347] Connected to a master server at 127.7.118.62:33763
I20260929 18:00:38.186813 21596 heartbeater.cc:464] Registering TS with master...
I20260929 18:00:38.187638 21596 heartbeater.cc:511] Master 127.7.118.62:33763 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.2@KRBTEST.COM" created.
I20260929 18:00:38.189577 21405 ts_manager.cc:194] Registered new tserver with Master: d0d030cb1647419cbf29b0991405c7ed (127.7.118.1:33205)
I20260929 18:00:38.191226 21405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.1@KRBTEST.COM'} at 127.7.118.1:40399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
I20260929 18:00:38.212651 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:0
--local_ip_for_outbound_sockets=127.7.118.2
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:33763
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:00:38.342453 21601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:38.342783 21601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:38.342839 21601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:00:38.347837 21601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:38.347935 21601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:38.348011 21601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 18:00:38.356098 21601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:33763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:38.357636 21601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:38.358896 21601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:00:38.370311 21607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:38.370359 21601 server_base.cc:1034] running on GCE node
W20260929 18:00:38.370244 21609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:38.370283 21606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:38.371093 21601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:38.372114 21601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:38.373391 21601 hybrid_clock.cc:648] HybridClock initialized: now 1790704838373343 us; error 67 us; skew 500 ppm
Sep 29 18:00:38 dist-test-slave-r7ph krb5kdc[21360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704838, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:38.377101 21601 init.cc:377] Logged in from keytab as kudu/127.7.118.2@KRBTEST.COM (short username kudu)
I20260929 18:00:38.378212 21601 webserver.cc:460] Webserver started at http://127.7.118.2:42555/ using document root <none> and password file <none>
I20260929 18:00:38.379027 21601 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:38.379104 21601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:00:38.379351 21601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:00:38.381935 21601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/data/instance:
uuid: "b889ace27b9f4ee18982909cd1d47114"
format_stamp: "Formatted at 2026-09-29 18:00:38 on dist-test-slave-r7ph"
server_key: "c6e2f7f6df0473c313c2cd4fa86f1063"
server_key_iv: "9af37916a4a1546e2e24f16376f62967"
server_key_version: "encryptionkey@0"
I20260929 18:00:38.382656 21601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance:
uuid: "b889ace27b9f4ee18982909cd1d47114"
format_stamp: "Formatted at 2026-09-29 18:00:38 on dist-test-slave-r7ph"
server_key: "c6e2f7f6df0473c313c2cd4fa86f1063"
server_key_iv: "9af37916a4a1546e2e24f16376f62967"
server_key_version: "encryptionkey@0"
I20260929 18:00:38.387184 21601 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 18:00:38.390151 21616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:38.391727 21601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:00:38.391906 21601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "b889ace27b9f4ee18982909cd1d47114"
format_stamp: "Formatted at 2026-09-29 18:00:38 on dist-test-slave-r7ph"
server_key: "c6e2f7f6df0473c313c2cd4fa86f1063"
server_key_iv: "9af37916a4a1546e2e24f16376f62967"
server_key_version: "encryptionkey@0"
I20260929 18:00:38.392088 21601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:38.415877 21601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:38.419821 21601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:38.420055 21601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:38.420759 21601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:00:38.421821 21601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:00:38.421911 21601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:38.422051 21601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:00:38.422086 21601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:38.432273 21601 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:44891
I20260929 18:00:38.432310 21729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:44891 every 8 connection(s)
I20260929 18:00:38.433290 21601 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 18:00:38.441169 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 21601
I20260929 18:00:38.441308 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance
I20260929 18:00:38.441586 7640 external_mini_cluster.cc:1428] Setting key ecc8dddcf52e59e939e8e76582453a49
Sep 29 18:00:38 dist-test-slave-r7ph krb5kdc[21360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704838, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:00:38.450126 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:38.435475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:40261 (local address 127.7.118.62:33763)
0929 18:00:38.435628 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:38.435633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:38.436656 (+ 1023us) server_negotiation.cc:407] Connection header received
0929 18:00:38.438270 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:38.438274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:38.438342 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:38.438423 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:38.439734 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:38.440285 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:38.441866 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:38.442221 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:38.442394 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:38.445508 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:38.445513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:38.445518 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:38.445565 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:38.447475 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:38.448170 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:38.448177 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:38.448181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:38.448289 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:38.448687 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:38.448694 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:38.448699 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:38.449011 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:38.449127 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:38.449757 (+ 630us) server_negotiation.cc:299] Negotiation successful
0929 18:00:38.449923 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 18:00:38.451092 21732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:38.435947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.2:40261)
0929 18:00:38.436465 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:38.436518 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:38.437972 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:38.438620 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:38.438629 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:38.439036 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:00:38.439563 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:38.439577 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:38.440434 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:38.440438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:38.441714 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:38.441727 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:38.442511 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:38.442517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:38.442694 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:38.443402 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:00:38.443420 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:00:38.445285 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:00:38.447639 (+ 2354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:38.447644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:38.447660 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:38.448044 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:38.448414 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:38.448417 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:38.448418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:38.448581 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:38.449159 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:00:38.449165 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:00:38.449480 (+ 315us) client_negotiation.cc:769] Sending connection context
0929 18:00:38.449665 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 18:00:38.449992 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":133,"threads_started":1}
WARNING: no policy specified for kudu/127.7.118.3@KRBTEST.COM; defaulting to no policy
I20260929 18:00:38.452555 21730 heartbeater.cc:347] Connected to a master server at 127.7.118.62:33763
I20260929 18:00:38.452817 21730 heartbeater.cc:464] Registering TS with master...
I20260929 18:00:38.453338 21730 heartbeater.cc:511] Master 127.7.118.62:33763 requested a full tablet report, sending...
I20260929 18:00:38.454686 21405 ts_manager.cc:194] Registered new tserver with Master: b889ace27b9f4ee18982909cd1d47114 (127.7.118.2:44891)
I20260929 18:00:38.455304 21405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.2@KRBTEST.COM'} at 127.7.118.2:40261
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
I20260929 18:00:38.475327 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:0
--local_ip_for_outbound_sockets=127.7.118.3
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:33763
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:00:38.594794 21735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:38.595119 21735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:38.595165 21735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:00:38.599664 21735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:38.599749 21735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:38.599817 21735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 18:00:38.606197 21735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:33763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:38.607460 21735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:38.608529 21735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:00:38.619800 21741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:38.619917 21743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:38.619822 21740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:38.620186 21735 server_base.cc:1034] running on GCE node
I20260929 18:00:38.620697 21735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:38.621460 21735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:38.622720 21735 hybrid_clock.cc:648] HybridClock initialized: now 1790704838622682 us; error 61 us; skew 500 ppm
Sep 29 18:00:38 dist-test-slave-r7ph krb5kdc[21360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704838, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:38.626137 21735 init.cc:377] Logged in from keytab as kudu/127.7.118.3@KRBTEST.COM (short username kudu)
I20260929 18:00:38.627290 21735 webserver.cc:460] Webserver started at http://127.7.118.3:35515/ using document root <none> and password file <none>
I20260929 18:00:38.627898 21735 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:38.627967 21735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:00:38.628147 21735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:00:38.630580 21735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/data/instance:
uuid: "596cf94ea8b44360adf9111ca5ba8485"
format_stamp: "Formatted at 2026-09-29 18:00:38 on dist-test-slave-r7ph"
server_key: "4403b0b34a242423303ec92dd8bd9ed1"
server_key_iv: "b61535a0707a6455a9b16118ea08af1e"
server_key_version: "encryptionkey@0"
I20260929 18:00:38.631196 21735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance:
uuid: "596cf94ea8b44360adf9111ca5ba8485"
format_stamp: "Formatted at 2026-09-29 18:00:38 on dist-test-slave-r7ph"
server_key: "4403b0b34a242423303ec92dd8bd9ed1"
server_key_iv: "b61535a0707a6455a9b16118ea08af1e"
server_key_version: "encryptionkey@0"
I20260929 18:00:38.635682 21735 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:00:38.639024 21750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:38.640628 21735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:00:38.640762 21735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "596cf94ea8b44360adf9111ca5ba8485"
format_stamp: "Formatted at 2026-09-29 18:00:38 on dist-test-slave-r7ph"
server_key: "4403b0b34a242423303ec92dd8bd9ed1"
server_key_iv: "b61535a0707a6455a9b16118ea08af1e"
server_key_version: "encryptionkey@0"
I20260929 18:00:38.640949 21735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:38.669303 21735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:38.673553 21735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:38.673772 21735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:38.674489 21735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:00:38.675441 21735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:00:38.675503 21735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:38.675551 21735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:00:38.675570 21735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:38.685985 21735 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:40605
I20260929 18:00:38.686003 21863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:40605 every 8 connection(s)
I20260929 18:00:38.687129 21735 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 18:00:38.693477 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 21735
I20260929 18:00:38.693662 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance
I20260929 18:00:38.694052 7640 external_mini_cluster.cc:1428] Setting key 6e299a99600e0e091a14e307f297b4fb
Sep 29 18:00:38 dist-test-slave-r7ph krb5kdc[21360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704838, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:00:38.704334 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:38.689278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:39823 (local address 127.7.118.62:33763)
0929 18:00:38.689469 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:38.689475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:38.690612 (+ 1137us) server_negotiation.cc:407] Connection header received
0929 18:00:38.692104 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:38.692107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:38.692170 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:38.692255 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:38.694034 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:38.694811 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:38.696379 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:38.696645 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:38.696980 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:38.700040 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:38.700043 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:38.700045 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:38.700075 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:38.701752 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:38.702472 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:38.702476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:38.702477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:38.702569 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:38.703052 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:38.703055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:38.703057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:38.703269 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:38.703368 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:38.704073 (+ 705us) server_negotiation.cc:299] Negotiation successful
0929 18:00:38.704203 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 18:00:38.705312 21866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:38.689786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.3:39823)
0929 18:00:38.690406 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:38.690471 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:38.691823 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:38.692471 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:38.692481 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:38.692911 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:00:38.693784 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:38.693815 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:38.694957 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:38.694961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:38.696236 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:38.696248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:38.697028 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:38.697032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:38.697300 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:38.698046 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:00:38.698065 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:00:38.699755 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:00:38.701927 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:38.701934 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:38.701950 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:38.702332 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:38.702688 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:38.702690 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:38.702692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:38.702916 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:38.703390 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:00:38.703396 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:00:38.703717 (+ 321us) client_negotiation.cc:769] Sending connection context
0929 18:00:38.703893 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 18:00:38.704222 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":143,"threads_started":1}
I20260929 18:00:38.706905 21864 heartbeater.cc:347] Connected to a master server at 127.7.118.62:33763
I20260929 18:00:38.707191 21864 heartbeater.cc:464] Registering TS with master...
I20260929 18:00:38.707858 21864 heartbeater.cc:511] Master 127.7.118.62:33763 requested a full tablet report, sending...
I20260929 18:00:38.708921 21405 ts_manager.cc:194] Registered new tserver with Master: 596cf94ea8b44360adf9111ca5ba8485 (127.7.118.3:40605)
I20260929 18:00:38.709525 21405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.3@KRBTEST.COM'} at 127.7.118.3:39823
I20260929 18:00:38.721135 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:00:38.731973 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:38.723424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32902 (local address 127.7.118.62:33763)
0929 18:00:38.723621 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:38.723629 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:38.723725 (+ 96us) server_negotiation.cc:407] Connection header received
0929 18:00:38.723829 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:38.723832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:38.723876 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:38.723951 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:38.724715 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:38.725271 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:38.726356 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:38.726622 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:38.726714 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:38.728311 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:38.728315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:38.728318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:38.728350 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:38.729959 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:38.730549 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:38.730552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:38.730553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:38.730641 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:38.730980 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:38.730982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:38.730983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:38.731237 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:38.731348 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:38.731646 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 18:00:38.731763 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 18:00:38.735910 21405 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:00:38.739460 21405 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:00:38.758234 21879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:38.751750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43056 (local address 127.7.118.3:40605)
0929 18:00:38.752563 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:38.752577 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:38.752604 (+ 27us) server_negotiation.cc:407] Connection header received
0929 18:00:38.752687 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:38.752694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:38.752916 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:38.753097 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:00:38.753323 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:38.754568 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:38.756417 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:38.757281 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:38.757417 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:38.757587 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:38.757953 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 18:00:38.758058 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":165,"threads_started":1}
I20260929 18:00:38.758230 21877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:38.751689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55274 (local address 127.7.118.1:33205)
0929 18:00:38.751956 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:38.751973 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:38.752386 (+ 413us) server_negotiation.cc:407] Connection header received
0929 18:00:38.752483 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:38.752490 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:38.752720 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:38.752903 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:00:38.753235 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:38.754485 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:38.756294 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:38.757281 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:38.757416 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:38.757587 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:38.757934 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 18:00:38.758058 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":84,"threads_started":1}
I20260929 18:00:38.758275 21875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:38.750951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:40605 (local address 127.0.0.1:43056)
0929 18:00:38.751379 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:38.751445 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:38.751610 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:38.753072 (+ 1462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:38.753075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:38.753094 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:00:38.753228 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:38.753235 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:38.754786 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:38.754790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:38.756283 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:38.756293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:38.757489 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:38.757493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:38.757626 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:38.757685 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 18:00:38.757923 (+ 238us) client_negotiation.cc:240] Negotiation successful
0929 18:00:38.758081 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":27,"thread_start_us":96,"threads_started":1}
I20260929 18:00:38.758656 21878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:38.751781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:44891 (local address 127.0.0.1:58168)
0929 18:00:38.752201 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:38.752218 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:38.752311 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:38.753725 (+ 1414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:38.753729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:38.753767 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:00:38.753869 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:38.753875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:38.755482 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:38.755485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:38.756596 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:38.756603 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:38.758167 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:38.758171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:38.758283 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:38.758353 (+ 70us) client_negotiation.cc:769] Sending connection context
0929 18:00:38.758433 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 18:00:38.758512 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":94,"threads_started":1}
I20260929 18:00:38.759094 21880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:38.752233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58168 (local address 127.7.118.2:44891)
0929 18:00:38.753225 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:38.753237 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:38.753258 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:00:38.753355 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:38.753360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:38.753552 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:38.753701 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:00:38.754024 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:38.755259 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:38.756829 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:38.757814 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:38.757944 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:38.758101 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:38.758839 (+ 738us) server_negotiation.cc:299] Negotiation successful
0929 18:00:38.758956 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":878,"thread_start_us":163,"threads_started":1}
I20260929 18:00:38.759099 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:38.751146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:33205 (local address 127.0.0.1:55274)
0929 18:00:38.752146 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:38.752162 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:38.752264 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:38.752893 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:38.752899 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:38.752922 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:00:38.753093 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:38.753103 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:38.754611 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:38.754615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:38.756159 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:38.756168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:38.757656 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:38.757660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:38.757744 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:38.757763 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 18:00:38.758844 (+ 1081us) client_negotiation.cc:240] Negotiation successful
0929 18:00:38.758953 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"mutex_wait_us":94,"thread_start_us":109,"threads_started":1}
I20260929 18:00:38.762019 21664 tablet_service.cc:1467] Processing CreateTablet for tablet d56923e0154f4a67b44f62216942c4e7 (DEFAULT_TABLE table=test-table [id=3aea378fa6334fb58d61399c5bb83f8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:00:38.762130 21529 tablet_service.cc:1467] Processing CreateTablet for tablet d56923e0154f4a67b44f62216942c4e7 (DEFAULT_TABLE table=test-table [id=3aea378fa6334fb58d61399c5bb83f8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:00:38.762127 21798 tablet_service.cc:1467] Processing CreateTablet for tablet d56923e0154f4a67b44f62216942c4e7 (DEFAULT_TABLE table=test-table [id=3aea378fa6334fb58d61399c5bb83f8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:00:38.763392 21664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d56923e0154f4a67b44f62216942c4e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:00:38.763515 21529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d56923e0154f4a67b44f62216942c4e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:00:38.763515 21798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d56923e0154f4a67b44f62216942c4e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:00:38.770994 21881 tablet_bootstrap.cc:492] T d56923e0154f4a67b44f62216942c4e7 P b889ace27b9f4ee18982909cd1d47114: Bootstrap starting.
I20260929 18:00:38.771773 21882 tablet_bootstrap.cc:492] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed: Bootstrap starting.
I20260929 18:00:38.774521 21882 tablet_bootstrap.cc:654] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed: Neither blocks nor log segments found. Creating new log.
I20260929 18:00:38.774940 21881 tablet_bootstrap.cc:654] T d56923e0154f4a67b44f62216942c4e7 P b889ace27b9f4ee18982909cd1d47114: Neither blocks nor log segments found. Creating new log.
I20260929 18:00:38.775836 21882 log.cc:824] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed: Log is configured to *not* fsync() on all Append() calls
I20260929 18:00:38.775954 21881 log.cc:824] T d56923e0154f4a67b44f62216942c4e7 P b889ace27b9f4ee18982909cd1d47114: Log is configured to *not* fsync() on all Append() calls
I20260929 18:00:38.776751 21883 tablet_bootstrap.cc:492] T d56923e0154f4a67b44f62216942c4e7 P 596cf94ea8b44360adf9111ca5ba8485: Bootstrap starting.
I20260929 18:00:38.778609 21882 tablet_bootstrap.cc:492] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed: No bootstrap required, opened a new log
I20260929 18:00:38.778847 21882 ts_tablet_manager.cc:1397] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 18:00:38.779074 21881 tablet_bootstrap.cc:492] T d56923e0154f4a67b44f62216942c4e7 P b889ace27b9f4ee18982909cd1d47114: No bootstrap required, opened a new log
I20260929 18:00:38.779307 21881 ts_tablet_manager.cc:1397] T d56923e0154f4a67b44f62216942c4e7 P b889ace27b9f4ee18982909cd1d47114: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 18:00:38.779429 21883 tablet_bootstrap.cc:654] T d56923e0154f4a67b44f62216942c4e7 P 596cf94ea8b44360adf9111ca5ba8485: Neither blocks nor log segments found. Creating new log.
I20260929 18:00:38.780880 21883 log.cc:824] T d56923e0154f4a67b44f62216942c4e7 P 596cf94ea8b44360adf9111ca5ba8485: Log is configured to *not* fsync() on all Append() calls
I20260929 18:00:38.783529 21883 tablet_bootstrap.cc:492] T d56923e0154f4a67b44f62216942c4e7 P 596cf94ea8b44360adf9111ca5ba8485: No bootstrap required, opened a new log
I20260929 18:00:38.783295 21881 raft_consensus.cc:348] T d56923e0154f4a67b44f62216942c4e7 P b889ace27b9f4ee18982909cd1d47114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } } peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } }
I20260929 18:00:38.783758 21883 ts_tablet_manager.cc:1397] T d56923e0154f4a67b44f62216942c4e7 P 596cf94ea8b44360adf9111ca5ba8485: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 18:00:38.783872 21881 raft_consensus.cc:374] T d56923e0154f4a67b44f62216942c4e7 P b889ace27b9f4ee18982909cd1d47114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:00:38.783946 21881 raft_consensus.cc:729] T d56923e0154f4a67b44f62216942c4e7 P b889ace27b9f4ee18982909cd1d47114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b889ace27b9f4ee18982909cd1d47114, State: Initialized, Role: FOLLOWER
I20260929 18:00:38.783741 21882 raft_consensus.cc:348] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } } peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } }
I20260929 18:00:38.784024 21882 raft_consensus.cc:374] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:00:38.784089 21882 raft_consensus.cc:729] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0d030cb1647419cbf29b0991405c7ed, State: Initialized, Role: FOLLOWER
I20260929 18:00:38.784487 21881 consensus_queue.cc:260] T d56923e0154f4a67b44f62216942c4e7 P b889ace27b9f4ee18982909cd1d47114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } } peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } }
I20260929 18:00:38.784595 21882 consensus_queue.cc:260] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } } peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } }
I20260929 18:00:38.785333 21730 heartbeater.cc:503] Master 127.7.118.62:33763 was elected leader, sending a full tablet report...
I20260929 18:00:38.785898 21881 ts_tablet_manager.cc:1428] T d56923e0154f4a67b44f62216942c4e7 P b889ace27b9f4ee18982909cd1d47114: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 18:00:38.787220 21882 ts_tablet_manager.cc:1428] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260929 18:00:38.787667 21596 heartbeater.cc:503] Master 127.7.118.62:33763 was elected leader, sending a full tablet report...
I20260929 18:00:38.788275 21883 raft_consensus.cc:348] T d56923e0154f4a67b44f62216942c4e7 P 596cf94ea8b44360adf9111ca5ba8485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } } peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } }
I20260929 18:00:38.788534 21883 raft_consensus.cc:374] T d56923e0154f4a67b44f62216942c4e7 P 596cf94ea8b44360adf9111ca5ba8485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:00:38.788569 21883 raft_consensus.cc:729] T d56923e0154f4a67b44f62216942c4e7 P 596cf94ea8b44360adf9111ca5ba8485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596cf94ea8b44360adf9111ca5ba8485, State: Initialized, Role: FOLLOWER
I20260929 18:00:38.789067 21883 consensus_queue.cc:260] T d56923e0154f4a67b44f62216942c4e7 P 596cf94ea8b44360adf9111ca5ba8485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } } peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } }
I20260929 18:00:38.789837 21864 heartbeater.cc:503] Master 127.7.118.62:33763 was elected leader, sending a full tablet report...
I20260929 18:00:38.790333 21883 ts_tablet_manager.cc:1428] T d56923e0154f4a67b44f62216942c4e7 P 596cf94ea8b44360adf9111ca5ba8485: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260929 18:00:38.916723 21597 tablet.cc:2369] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:00:38.934595 21731 tablet.cc:2369] T d56923e0154f4a67b44f62216942c4e7 P b889ace27b9f4ee18982909cd1d47114: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:00:38.938339 21865 tablet.cc:2369] T d56923e0154f4a67b44f62216942c4e7 P 596cf94ea8b44360adf9111ca5ba8485: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:00:39.037725 21888 raft_consensus.cc:482] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:00:39.038031 21888 raft_consensus.cc:504] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } } peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } }
I20260929 18:00:39.039613 21888 leader_election.cc:288] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 596cf94ea8b44360adf9111ca5ba8485 (127.7.118.3:40605), b889ace27b9f4ee18982909cd1d47114 (127.7.118.2:44891)
I20260929 18:00:39.047183 21880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:39.040419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:37119 (local address 127.7.118.2:44891)
0929 18:00:39.040565 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:39.040571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:39.040612 (+ 41us) server_negotiation.cc:407] Connection header received
0929 18:00:39.040676 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:39.040680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:39.040754 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:39.040848 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:00:39.042381 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:39.043422 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:39.045783 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:39.046459 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:39.046570 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:39.046626 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:39.046946 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 18:00:39.047048 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:00:39.047212 21879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:39.039965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:49789 (local address 127.7.118.3:40605)
0929 18:00:39.040104 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:39.040109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:39.040372 (+ 263us) server_negotiation.cc:407] Connection header received
0929 18:00:39.040609 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:39.040615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:39.040680 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:39.040720 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:00:39.041269 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:39.042183 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:39.045760 (+ 3577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:39.046459 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:39.046554 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:39.046626 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:39.047025 (+ 399us) server_negotiation.cc:299] Negotiation successful
0929 18:00:39.047106 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:00:39.047243 21891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:39.040010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:44891 (local address 127.7.118.1:37119)
0929 18:00:39.040450 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:39.040463 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:39.040539 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:39.040876 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:39.040879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:39.040939 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:00:39.041127 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:39.041135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:39.043642 (+ 2507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:39.043646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:39.045575 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:39.045593 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:39.046607 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:39.046614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:39.046763 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:39.046799 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 18:00:39.046899 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 18:00:39.047007 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":210,"thread_start_us":107,"threads_started":1}
I20260929 18:00:39.047626 21890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:39.039887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:40605 (local address 127.7.118.1:49789)
0929 18:00:39.040268 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:39.040284 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:39.040464 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:39.040876 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:39.040879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:39.040920 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:00:39.041128 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:39.041135 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:39.043716 (+ 2581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:39.043722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:39.045575 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:39.045593 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:39.046734 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:39.046736 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:39.046846 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:39.046892 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 18:00:39.047382 (+ 490us) client_negotiation.cc:240] Negotiation successful
0929 18:00:39.047484 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":157,"spinlock_wait_cycles":32512,"thread_start_us":117,"threads_started":1}
I20260929 18:00:39.048298 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d56923e0154f4a67b44f62216942c4e7" candidate_uuid: "d0d030cb1647419cbf29b0991405c7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b889ace27b9f4ee18982909cd1d47114" is_pre_election: true
I20260929 18:00:39.048513 21818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d56923e0154f4a67b44f62216942c4e7" candidate_uuid: "d0d030cb1647419cbf29b0991405c7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596cf94ea8b44360adf9111ca5ba8485" is_pre_election: true
I20260929 18:00:39.048729 21684 raft_consensus.cc:2454] T d56923e0154f4a67b44f62216942c4e7 P b889ace27b9f4ee18982909cd1d47114 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0d030cb1647419cbf29b0991405c7ed in term 0.
I20260929 18:00:39.048846 21818 raft_consensus.cc:2454] T d56923e0154f4a67b44f62216942c4e7 P 596cf94ea8b44360adf9111ca5ba8485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0d030cb1647419cbf29b0991405c7ed in term 0.
I20260929 18:00:39.049302 21484 leader_election.cc:302] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b889ace27b9f4ee18982909cd1d47114, d0d030cb1647419cbf29b0991405c7ed; no voters:
I20260929 18:00:39.049639 21888 raft_consensus.cc:2781] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:00:39.049732 21888 raft_consensus.cc:482] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:00:39.049775 21888 raft_consensus.cc:3037] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:00:39.051208 21888 raft_consensus.cc:504] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } } peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } }
I20260929 18:00:39.051656 21888 leader_election.cc:288] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 1 election: Requested vote from peers 596cf94ea8b44360adf9111ca5ba8485 (127.7.118.3:40605), b889ace27b9f4ee18982909cd1d47114 (127.7.118.2:44891)
I20260929 18:00:39.052119 21818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d56923e0154f4a67b44f62216942c4e7" candidate_uuid: "d0d030cb1647419cbf29b0991405c7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596cf94ea8b44360adf9111ca5ba8485"
I20260929 18:00:39.052296 21818 raft_consensus.cc:3037] T d56923e0154f4a67b44f62216942c4e7 P 596cf94ea8b44360adf9111ca5ba8485 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:00:39.054008 21818 raft_consensus.cc:2454] T d56923e0154f4a67b44f62216942c4e7 P 596cf94ea8b44360adf9111ca5ba8485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0d030cb1647419cbf29b0991405c7ed in term 1.
I20260929 18:00:39.054471 21684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d56923e0154f4a67b44f62216942c4e7" candidate_uuid: "d0d030cb1647419cbf29b0991405c7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b889ace27b9f4ee18982909cd1d47114"
I20260929 18:00:39.054625 21684 raft_consensus.cc:3037] T d56923e0154f4a67b44f62216942c4e7 P b889ace27b9f4ee18982909cd1d47114 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:00:39.054673 21486 leader_election.cc:302] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 596cf94ea8b44360adf9111ca5ba8485, d0d030cb1647419cbf29b0991405c7ed; no voters:
I20260929 18:00:39.054919 21888 raft_consensus.cc:2781] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:00:39.055162 21888 raft_consensus.cc:686] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed [term 1 LEADER]: Becoming Leader. State: Replica: d0d030cb1647419cbf29b0991405c7ed, State: Running, Role: LEADER
I20260929 18:00:39.055533 21888 consensus_queue.cc:237] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } } peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } }
I20260929 18:00:39.056813 21684 raft_consensus.cc:2454] T d56923e0154f4a67b44f62216942c4e7 P b889ace27b9f4ee18982909cd1d47114 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0d030cb1647419cbf29b0991405c7ed in term 1.
I20260929 18:00:39.059150 21402 catalog_manager.cc:5495] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to d0d030cb1647419cbf29b0991405c7ed (127.7.118.1). New cstate: current_term: 1 leader_uuid: "d0d030cb1647419cbf29b0991405c7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } health_report { overall_health: UNKNOWN } } }
I20260929 18:00:39.127524 21877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:39.123306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55282 (local address 127.7.118.1:33205)
0929 18:00:39.123447 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:39.123457 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:39.123471 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:00:39.123553 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:39.123556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:39.123667 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:39.123964 (+ 297us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:00:39.124260 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:39.124885 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:39.125823 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:39.126053 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:39.126141 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:39.126611 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:00:39.127094 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:00:39.127204 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:39.127340 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 18:00:39.127402 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:00:39.134552 21684 raft_consensus.cc:1261] T d56923e0154f4a67b44f62216942c4e7 P b889ace27b9f4ee18982909cd1d47114 [term 1 FOLLOWER]: Refusing update from remote peer d0d030cb1647419cbf29b0991405c7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:00:39.134552 21818 raft_consensus.cc:1261] T d56923e0154f4a67b44f62216942c4e7 P 596cf94ea8b44360adf9111ca5ba8485 [term 1 FOLLOWER]: Refusing update from remote peer d0d030cb1647419cbf29b0991405c7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:00:39.135456 21888 consensus_queue.cc:1035] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:00:39.135740 21892 consensus_queue.cc:1035] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:00:39.146641 21895 mvcc.cc:205] Tried to move back new op lower bound from 7334727021085736960 to 7334727020775501824. Current Snapshot: MvccSnapshot[applied={T|T < 7334727021085736960}]
I20260929 18:00:39.148444 21896 mvcc.cc:205] Tried to move back new op lower bound from 7334727021085736960 to 7334727020775501824. Current Snapshot: MvccSnapshot[applied={T|T < 7334727021085736960}]
I20260929 18:00:39.148448 21898 mvcc.cc:205] Tried to move back new op lower bound from 7334727021085736960 to 7334727020775501824. Current Snapshot: MvccSnapshot[applied={T|T < 7334727021085736960}]
I20260929 18:00:39.157521 21405 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:00:39.157702 21405 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:00:39.160622 21405 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627: Sending DeleteTablet for 3 replicas of tablet d56923e0154f4a67b44f62216942c4e7
I20260929 18:00:39.162091 21529 tablet_service.cc:1514] Processing DeleteTablet for tablet d56923e0154f4a67b44f62216942c4e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:00:39 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:55274
I20260929 18:00:39.162092 21798 tablet_service.cc:1514] Processing DeleteTablet for tablet d56923e0154f4a67b44f62216942c4e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:00:39 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:43056
I20260929 18:00:39.162092 21664 tablet_service.cc:1514] Processing DeleteTablet for tablet d56923e0154f4a67b44f62216942c4e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:00:39 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:58168
I20260929 18:00:39.162786 21905 tablet_replica.cc:330] T d56923e0154f4a67b44f62216942c4e7 P 596cf94ea8b44360adf9111ca5ba8485: stopping tablet replica
I20260929 18:00:39.162833 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 21467
I20260929 18:00:39.164250 21904 tablet_replica.cc:330] T d56923e0154f4a67b44f62216942c4e7 P b889ace27b9f4ee18982909cd1d47114: stopping tablet replica
I20260929 18:00:39.164561 21905 raft_consensus.cc:2229] T d56923e0154f4a67b44f62216942c4e7 P 596cf94ea8b44360adf9111ca5ba8485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:00:39.164644 21904 raft_consensus.cc:2229] T d56923e0154f4a67b44f62216942c4e7 P b889ace27b9f4ee18982909cd1d47114 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:00:39.164964 21905 raft_consensus.cc:2258] T d56923e0154f4a67b44f62216942c4e7 P 596cf94ea8b44360adf9111ca5ba8485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:00:39.164968 21904 raft_consensus.cc:2258] T d56923e0154f4a67b44f62216942c4e7 P b889ace27b9f4ee18982909cd1d47114 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:00:39.166613 21904 ts_tablet_manager.cc:1905] T d56923e0154f4a67b44f62216942c4e7 P b889ace27b9f4ee18982909cd1d47114: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:00:39.169503 21904 ts_tablet_manager.cc:1918] T d56923e0154f4a67b44f62216942c4e7 P b889ace27b9f4ee18982909cd1d47114: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:00:39.169772 21904 log.cc:1196] T d56923e0154f4a67b44f62216942c4e7 P b889ace27b9f4ee18982909cd1d47114: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/wal/wals/d56923e0154f4a67b44f62216942c4e7
I20260929 18:00:39.170226 21904 ts_tablet_manager.cc:1939] T d56923e0154f4a67b44f62216942c4e7 P b889ace27b9f4ee18982909cd1d47114: Deleting consensus metadata
I20260929 18:00:39.170431 21905 ts_tablet_manager.cc:1905] T d56923e0154f4a67b44f62216942c4e7 P 596cf94ea8b44360adf9111ca5ba8485: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:00:39.171386 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 21601
W20260929 18:00:39.171446 21390 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.118.1:33205 (error 108)
I20260929 18:00:39.173626 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:39.173226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:33205 (local address 127.0.0.1:55290)
0929 18:00:39.173446 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:39.173523 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.1:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260929 18:00:39.173866 21390 catalog_manager.cc:4584] TS d0d030cb1647419cbf29b0991405c7ed (127.7.118.1:33205): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d56923e0154f4a67b44f62216942c4e7: Network error: Client connection negotiation failed: client connection to 127.7.118.1:33205: connect: Connection refused (error 111)
I20260929 18:00:39.174598 21905 ts_tablet_manager.cc:1918] T d56923e0154f4a67b44f62216942c4e7 P 596cf94ea8b44360adf9111ca5ba8485: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:00:39.174697 21905 log.cc:1196] T d56923e0154f4a67b44f62216942c4e7 P 596cf94ea8b44360adf9111ca5ba8485: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/wal/wals/d56923e0154f4a67b44f62216942c4e7
I20260929 18:00:39.175032 21905 ts_tablet_manager.cc:1939] T d56923e0154f4a67b44f62216942c4e7 P 596cf94ea8b44360adf9111ca5ba8485: Deleting consensus metadata
I20260929 18:00:39.176975 21392 catalog_manager.cc:4841] TS 596cf94ea8b44360adf9111ca5ba8485 (127.7.118.3:40605): tablet d56923e0154f4a67b44f62216942c4e7 (table test-table [id=3aea378fa6334fb58d61399c5bb83f8b]) successfully deleted
I20260929 18:00:39.178706 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 21735
I20260929 18:00:39.182657 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:39.178930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:44891 (local address 127.0.0.1:58172)
0929 18:00:39.182482 (+ 3552us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:39.182562 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.2:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3341}
I20260929 18:00:39.185034 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 21373
I20260929 18:00:39.192117 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:33763
--webserver_interface=127.7.118.62
--webserver_port=39109
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:33763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:00:39.312402 21908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:39.312733 21908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:39.312786 21908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:00:39.317222 21908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:00:39.317317 21908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:39.317337 21908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:39.317356 21908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:00:39.317373 21908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:00:39.323433 21908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:33763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:33763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=39109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:39.324632 21908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:39.325740 21908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:00:39.340013 21917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:39.339998 21915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:39.340137 21908 server_base.cc:1034] running on GCE node
W20260929 18:00:39.340005 21914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:39.341130 21908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:39.342630 21908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:39.343868 21908 hybrid_clock.cc:648] HybridClock initialized: now 1790704839343829 us; error 56 us; skew 500 ppm
Sep 29 18:00:39 dist-test-slave-r7ph krb5kdc[21360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704839, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:39.347448 21908 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 18:00:39.348623 21908 webserver.cc:460] Webserver started at http://127.7.118.62:39109/ using document root <none> and password file <none>
I20260929 18:00:39.349249 21908 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:39.349337 21908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:00:39.353938 21908 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:00:39.356690 21924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:39.358238 21908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 18:00:39.358495 21908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "8a24a1dd183f432da92e1a3f81726627"
format_stamp: "Formatted at 2026-09-29 18:00:37 on dist-test-slave-r7ph"
server_key: "3e5f2ab3a17cbd435f56dd7f3c28236d"
server_key_iv: "9c208ea10dd20b632694d814569798ce"
server_key_version: "encryptionkey@0"
I20260929 18:00:39.359194 21908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:39.371904 21908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:39.375818 21908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:39.376011 21908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:39.383656 21908 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:33763
I20260929 18:00:39.384037 21976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:33763 every 8 connection(s)
I20260929 18:00:39.384977 21908 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:00:39.390003 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 21908
I20260929 18:00:39.390094 21977 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:00:39.390532 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:33205
--local_ip_for_outbound_sockets=127.7.118.1
--tserver_master_addrs=127.7.118.62:33763
--webserver_port=38141
--webserver_interface=127.7.118.1
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:00:39.392535 21977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627: Bootstrap starting.
I20260929 18:00:39.411252 21977 log.cc:824] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627: Log is configured to *not* fsync() on all Append() calls
I20260929 18:00:39.421447 21977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:00:39.421926 21977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627: Bootstrap complete.
I20260929 18:00:39.426774 21977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a24a1dd183f432da92e1a3f81726627" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33763 } }
I20260929 18:00:39.427451 21977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a24a1dd183f432da92e1a3f81726627, State: Initialized, Role: FOLLOWER
I20260929 18:00:39.428357 21977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a24a1dd183f432da92e1a3f81726627" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33763 } }
I20260929 18:00:39.428510 21977 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:00:39.428640 21977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:00:39.428753 21977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:00:39.431180 21977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a24a1dd183f432da92e1a3f81726627" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33763 } }
I20260929 18:00:39.431610 21977 leader_election.cc:302] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a24a1dd183f432da92e1a3f81726627; no voters:
I20260929 18:00:39.432020 21977 leader_election.cc:288] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:00:39.432144 21981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:00:39.432369 21981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [term 2 LEADER]: Becoming Leader. State: Replica: 8a24a1dd183f432da92e1a3f81726627, State: Running, Role: LEADER
I20260929 18:00:39.432708 21981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a24a1dd183f432da92e1a3f81726627" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33763 } }
I20260929 18:00:39.433446 21977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:00:39.434448 21982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8a24a1dd183f432da92e1a3f81726627" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a24a1dd183f432da92e1a3f81726627" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33763 } } }
I20260929 18:00:39.434595 21982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [sys.catalog]: This master's current role is: LEADER
I20260929 18:00:39.434998 21990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:00:39.435032 21983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a24a1dd183f432da92e1a3f81726627. Latest consensus state: current_term: 2 leader_uuid: "8a24a1dd183f432da92e1a3f81726627" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a24a1dd183f432da92e1a3f81726627" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33763 } } }
I20260929 18:00:39.435150 21983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [sys.catalog]: This master's current role is: LEADER
I20260929 18:00:39.442417 21990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:00:39.443351 21990 catalog_manager.cc:1259] Loaded cluster ID: bb83e00edd5f49829d3c0e9d1eada7cc
I20260929 18:00:39.443441 21990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:00:39.444662 21990 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 18:00:39.447696 21990 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 18:00:39.452521 21990 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:00:39.453387 21990 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627: Loaded TSK: 0
I20260929 18:00:39.454527 21990 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:00:39.532246 21979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:39.532612 21979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:39.532694 21979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:00:39.537827 21979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:39.537946 21979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:39.538048 21979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 18:00:39.543991 21979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:33205
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=38141
--tserver_master_addrs=127.7.118.62:33763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:39.545186 21979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:39.546376 21979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:00:39.557500 22004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:39.557844 22007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:39.557500 22005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:39.558060 21979 server_base.cc:1034] running on GCE node
I20260929 18:00:39.558621 21979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:39.559388 21979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:39.560597 21979 hybrid_clock.cc:648] HybridClock initialized: now 1790704839560569 us; error 42 us; skew 500 ppm
Sep 29 18:00:39 dist-test-slave-r7ph krb5kdc[21360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704839, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:39.564005 21979 init.cc:377] Logged in from keytab as kudu/127.7.118.1@KRBTEST.COM (short username kudu)
I20260929 18:00:39.565150 21979 webserver.cc:460] Webserver started at http://127.7.118.1:38141/ using document root <none> and password file <none>
I20260929 18:00:39.565770 21979 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:39.565851 21979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:00:39.570430 21979 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:00:39.572928 22014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:39.574549 21979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:00:39.574681 21979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "d0d030cb1647419cbf29b0991405c7ed"
format_stamp: "Formatted at 2026-09-29 18:00:38 on dist-test-slave-r7ph"
server_key: "2d221812c87fbc319150cd86b28ad975"
server_key_iv: "3338f886ffdca4aa7f6513fb8a4716fa"
server_key_version: "encryptionkey@0"
I20260929 18:00:39.575351 21979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:39.592691 21979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:39.596928 21979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:39.597186 21979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:39.598073 21979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:00:39.599519 22021 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:00:39.604737 21979 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:00:39.604838 21979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260929 18:00:39.604897 21979 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:00:39.606976 21979 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:00:39.607060 21979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260929 18:00:39.607329 22021 tablet_bootstrap.cc:492] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed: Bootstrap starting.
I20260929 18:00:39.619220 21979 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:33205
I20260929 18:00:39.619266 22128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:33205 every 8 connection(s)
I20260929 18:00:39.620666 21979 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 18:00:39.629110 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 21979
I20260929 18:00:39.629590 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:44891
--local_ip_for_outbound_sockets=127.7.118.2
--tserver_master_addrs=127.7.118.62:33763
--webserver_port=42555
--webserver_interface=127.7.118.2
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:00:39.630419 22021 log.cc:824] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:00:39 dist-test-slave-r7ph krb5kdc[21360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704839, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:00:39.639983 22021 tablet_bootstrap.cc:492] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:00:39.640545 22021 tablet_bootstrap.cc:492] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed: Bootstrap complete.
I20260929 18:00:39.641827 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:39.623314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:38909 (local address 127.7.118.62:33763)
0929 18:00:39.623683 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:39.623705 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:39.624106 (+ 401us) server_negotiation.cc:407] Connection header received
0929 18:00:39.625333 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:39.625345 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:39.625680 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:39.626051 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:39.627325 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:39.628713 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:39.630198 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:39.630857 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:39.631108 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:39.634751 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:39.634758 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:39.634777 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:39.634817 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:39.637431 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:39.638294 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:39.638302 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:39.638308 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:39.638491 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:39.638971 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:39.638975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:39.638977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:39.639480 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:39.639623 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:39.640433 (+ 810us) server_negotiation.cc:299] Negotiation successful
0929 18:00:39.640818 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":135,"threads_started":1}
I20260929 18:00:39.641884 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:39.623283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.1:38909)
0929 18:00:39.623920 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:39.623974 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:39.624861 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:39.626115 (+ 1254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:39.626123 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:39.626612 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:00:39.627130 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:39.627146 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:39.628884 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:39.628888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:39.630047 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:39.630061 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:39.631295 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:39.631301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:39.631541 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:39.632320 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:00:39.632342 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:00:39.634473 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:00:39.637619 (+ 3146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:39.637629 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:39.637649 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:39.638110 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:39.638612 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:39.638617 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:39.638621 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:39.638843 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:39.639622 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:00:39.639632 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:00:39.640014 (+ 382us) client_negotiation.cc:769] Sending connection context
0929 18:00:39.640265 (+ 251us) client_negotiation.cc:240] Negotiation successful
0929 18:00:39.640715 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":182,"threads_started":1}
I20260929 18:00:39.642068 22021 ts_tablet_manager.cc:1397] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed: Time spent bootstrapping tablet: real 0.035s user 0.024s sys 0.009s
I20260929 18:00:39.645123 22129 heartbeater.cc:347] Connected to a master server at 127.7.118.62:33763
I20260929 18:00:39.645462 22129 heartbeater.cc:464] Registering TS with master...
I20260929 18:00:39.645123 22021 raft_consensus.cc:348] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } } peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } }
I20260929 18:00:39.645753 22021 raft_consensus.cc:729] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0d030cb1647419cbf29b0991405c7ed, State: Initialized, Role: FOLLOWER
I20260929 18:00:39.646098 22129 heartbeater.cc:511] Master 127.7.118.62:33763 requested a full tablet report, sending...
I20260929 18:00:39.646329 22021 consensus_queue.cc:260] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } } peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } }
I20260929 18:00:39.647526 22021 ts_tablet_manager.cc:1428] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:00:39.649314 21939 ts_manager.cc:194] Registered new tserver with Master: d0d030cb1647419cbf29b0991405c7ed (127.7.118.1:33205)
I20260929 18:00:39.653103 21939 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.1@KRBTEST.COM'} at 127.7.118.1:38909
I20260929 18:00:39.656327 22129 heartbeater.cc:503] Master 127.7.118.62:33763 was elected leader, sending a full tablet report...
Sep 29 18:00:39 dist-test-slave-r7ph krb5kdc[21360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704839, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for kudu/127.7.118.1@KRBTEST.COM
I20260929 18:00:39.666096 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:39.652613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:33205 (local address 127.0.0.1:55294)
0929 18:00:39.653645 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:39.653714 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:39.653892 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:39.654686 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:39.654692 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:39.655175 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:00:39.655317 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:39.655326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:39.656755 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:39.656760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:39.657715 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:39.657726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:39.658577 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:39.658580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:39.658682 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:39.659330 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:00:39.659343 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:00:39.661114 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:00:39.663659 (+ 2545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:39.663670 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:39.663675 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:39.663979 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:39.664515 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:39.664519 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:39.664521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:39.664664 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:39.665276 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:00:39.665282 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:00:39.665419 (+ 137us) client_negotiation.cc:769] Sending connection context
0929 18:00:39.665666 (+ 247us) client_negotiation.cc:240] Negotiation successful
0929 18:00:39.665929 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":127,"threads_started":1}
I20260929 18:00:39.666163 22137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:39.652744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55294 (local address 127.7.118.1:33205)
0929 18:00:39.653220 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:39.653235 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:39.653914 (+ 679us) server_negotiation.cc:407] Connection header received
0929 18:00:39.654141 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:39.654148 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:39.654499 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:39.654654 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:39.655465 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:39.656614 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:39.657860 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:39.658447 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:39.658560 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:39.661315 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:39.661332 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:39.661347 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:39.661382 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:39.663464 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:39.664213 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:39.664219 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:39.664221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:39.664351 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:39.664776 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:39.664779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:39.664781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:39.665128 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:39.665258 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:39.665774 (+ 516us) server_negotiation.cc:299] Negotiation successful
0929 18:00:39.665988 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":115,"threads_started":1}
I20260929 18:00:39.667582 22063 tablet_service.cc:1514] Processing DeleteTablet for tablet d56923e0154f4a67b44f62216942c4e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:00:39 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:55294
I20260929 18:00:39.668519 22138 tablet_replica.cc:330] stopping tablet replica
I20260929 18:00:39.668853 22138 raft_consensus.cc:2229] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:00:39.669122 22138 raft_consensus.cc:2258] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:00:39.670209 22138 ts_tablet_manager.cc:1905] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:00:39.674088 22138 ts_tablet_manager.cc:1918] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:00:39.674201 22138 log.cc:1196] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/wal/wals/d56923e0154f4a67b44f62216942c4e7
I20260929 18:00:39.674540 22138 ts_tablet_manager.cc:1939] T d56923e0154f4a67b44f62216942c4e7 P d0d030cb1647419cbf29b0991405c7ed: Deleting consensus metadata
I20260929 18:00:39.676474 21926 catalog_manager.cc:4841] TS d0d030cb1647419cbf29b0991405c7ed (127.7.118.1:33205): tablet d56923e0154f4a67b44f62216942c4e7 (table test-table [id=3aea378fa6334fb58d61399c5bb83f8b]) successfully deleted
W20260929 18:00:39.758910 22134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:39.759282 22134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:39.759377 22134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:00:39.763778 22134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:39.763885 22134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:39.763993 22134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 18:00:39.770109 22134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:44891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=42555
--tserver_master_addrs=127.7.118.62:33763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:39.771389 22134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:39.772490 22134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:00:39.783242 22144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:39.783262 22146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:39.783243 22143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:39.783816 22134 server_base.cc:1034] running on GCE node
I20260929 18:00:39.784245 22134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:39.784942 22134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:39.786198 22134 hybrid_clock.cc:648] HybridClock initialized: now 1790704839786158 us; error 69 us; skew 500 ppm
Sep 29 18:00:39 dist-test-slave-r7ph krb5kdc[21360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704839, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:39.789526 22134 init.cc:377] Logged in from keytab as kudu/127.7.118.2@KRBTEST.COM (short username kudu)
I20260929 18:00:39.790680 22134 webserver.cc:460] Webserver started at http://127.7.118.2:42555/ using document root <none> and password file <none>
I20260929 18:00:39.791256 22134 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:39.791344 22134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:00:39.795581 22134 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 18:00:39.798040 22153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:39.799543 22134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:00:39.799697 22134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "b889ace27b9f4ee18982909cd1d47114"
format_stamp: "Formatted at 2026-09-29 18:00:38 on dist-test-slave-r7ph"
server_key: "c6e2f7f6df0473c313c2cd4fa86f1063"
server_key_iv: "9af37916a4a1546e2e24f16376f62967"
server_key_version: "encryptionkey@0"
I20260929 18:00:39.800344 22134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:39.814128 22134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:39.818387 22134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:39.818615 22134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:39.819329 22134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:00:39.820281 22134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:00:39.820358 22134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:39.820432 22134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:00:39.820485 22134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:39.831646 22134 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:44891
I20260929 18:00:39.831669 22266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:44891 every 8 connection(s)
I20260929 18:00:39.832813 22134 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 18:00:39.838147 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 22134
I20260929 18:00:39.838663 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:40605
--local_ip_for_outbound_sockets=127.7.118.3
--tserver_master_addrs=127.7.118.62:33763
--webserver_port=35515
--webserver_interface=127.7.118.3
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:00:39 dist-test-slave-r7ph krb5kdc[21360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704839, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:00:39.850660 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:39.835050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:38997 (local address 127.7.118.62:33763)
0929 18:00:39.835202 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:39.835207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:39.836860 (+ 1653us) server_negotiation.cc:407] Connection header received
0929 18:00:39.837883 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:39.837889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:39.838021 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:39.838130 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:39.839720 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:39.840383 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:39.842178 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:39.842559 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:39.842698 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:39.845786 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:39.845790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:39.845792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:39.845823 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:39.847634 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:39.848324 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:39.848331 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:39.848337 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:39.848457 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:39.848942 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:39.848947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:39.848951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:39.849256 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:39.849348 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:39.850283 (+ 935us) server_negotiation.cc:299] Negotiation successful
0929 18:00:39.850476 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:00:39.851610 22269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:39.835579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.2:38997)
0929 18:00:39.836641 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:39.836692 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:39.837592 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:39.838454 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:39.838462 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:39.838986 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:00:39.839533 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:39.839548 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:39.840559 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:39.840567 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:39.842012 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:39.842025 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:39.842768 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:39.842773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:39.843082 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:39.843929 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:00:39.843949 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:00:39.845552 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:00:39.847831 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:39.847839 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:39.847861 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:39.848197 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:39.848570 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:39.848573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:39.848575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:39.848773 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:39.849462 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:00:39.849471 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:00:39.849967 (+ 496us) client_negotiation.cc:769] Sending connection context
0929 18:00:39.850204 (+ 237us) client_negotiation.cc:240] Negotiation successful
0929 18:00:39.850647 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"thread_start_us":590,"threads_started":1}
I20260929 18:00:39.853330 22267 heartbeater.cc:347] Connected to a master server at 127.7.118.62:33763
I20260929 18:00:39.853636 22267 heartbeater.cc:464] Registering TS with master...
I20260929 18:00:39.854337 22267 heartbeater.cc:511] Master 127.7.118.62:33763 requested a full tablet report, sending...
I20260929 18:00:39.855628 21939 ts_manager.cc:194] Registered new tserver with Master: b889ace27b9f4ee18982909cd1d47114 (127.7.118.2:44891)
I20260929 18:00:39.856539 21939 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.2@KRBTEST.COM'} at 127.7.118.2:38997
W20260929 18:00:39.962422 22270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:39.962744 22270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:39.962790 22270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:00:39.967293 22270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:39.967372 22270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:39.967438 22270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 18:00:39.973107 22270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:40605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=35515
--tserver_master_addrs=127.7.118.62:33763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:39.974323 22270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:39.975548 22270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:00:39.986764 22278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:39.986726 22276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:39.986795 22275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:39.986931 22270 server_base.cc:1034] running on GCE node
I20260929 18:00:39.987556 22270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:39.988410 22270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:39.989639 22270 hybrid_clock.cc:648] HybridClock initialized: now 1790704839989598 us; error 58 us; skew 500 ppm
Sep 29 18:00:39 dist-test-slave-r7ph krb5kdc[21360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704839, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:39.993000 22270 init.cc:377] Logged in from keytab as kudu/127.7.118.3@KRBTEST.COM (short username kudu)
I20260929 18:00:39.994174 22270 webserver.cc:460] Webserver started at http://127.7.118.3:35515/ using document root <none> and password file <none>
I20260929 18:00:39.994884 22270 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:39.995018 22270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:00:39.999552 22270 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:00:40.002104 22285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:40.003970 22270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:00:40.004165 22270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "596cf94ea8b44360adf9111ca5ba8485"
format_stamp: "Formatted at 2026-09-29 18:00:38 on dist-test-slave-r7ph"
server_key: "4403b0b34a242423303ec92dd8bd9ed1"
server_key_iv: "b61535a0707a6455a9b16118ea08af1e"
server_key_version: "encryptionkey@0"
I20260929 18:00:40.004858 22270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:40.027503 22270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:40.031620 22270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:40.031800 22270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:40.032521 22270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:00:40.033716 22270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:00:40.033782 22270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:40.033838 22270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:00:40.033854 22270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:40.044051 22270 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:40605
I20260929 18:00:40.044068 22398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:40605 every 8 connection(s)
I20260929 18:00:40.045186 22270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 18:00:40.046933 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 22270
Sep 29 18:00:40 dist-test-slave-r7ph krb5kdc[21360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704839, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:00:40.057049 22401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:40.047597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32906 (local address 127.7.118.62:33763)
0929 18:00:40.047913 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:40.047917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:40.048073 (+ 156us) server_negotiation.cc:407] Connection header received
0929 18:00:40.048183 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:40.048187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:40.048242 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:40.048359 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:40.049415 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:40.049977 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:40.051251 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:40.051503 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:40.051594 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:40.053203 (+ 1609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:40.053206 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:40.053208 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:40.053234 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:40.054959 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:40.055517 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:40.055520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:40.055522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:40.055622 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:40.055951 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:40.055955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:40.055956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:40.056167 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:40.056456 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:40.056658 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 18:00:40.056837 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":92,"threads_started":1}
I20260929 18:00:40.061550 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:40.047409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:44289 (local address 127.7.118.62:33763)
0929 18:00:40.047565 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:40.047570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:40.048895 (+ 1325us) server_negotiation.cc:407] Connection header received
0929 18:00:40.050546 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:40.050548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:40.050594 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:40.050664 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:40.052291 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:40.052871 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:40.054171 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:40.054485 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:40.054601 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:40.057373 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:40.057378 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:40.057381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:40.057408 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:40.059169 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:40.059848 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:40.059852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:40.059853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:40.059934 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:40.060341 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:40.060344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:40.060346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:40.060558 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:40.060645 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:40.061269 (+ 624us) server_negotiation.cc:299] Negotiation successful
0929 18:00:40.061409 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 18:00:40.062546 22403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:40.048010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.3:44289)
0929 18:00:40.048689 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:40.048754 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:40.050294 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:40.050877 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:40.050888 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:40.051398 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:00:40.052090 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:40.052108 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:40.053022 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:40.053026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:40.054032 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:40.054042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:40.054658 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:40.054662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:40.054856 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:40.055387 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:00:40.055411 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:00:40.057197 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:00:40.059327 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:40.059333 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:40.059353 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:40.059728 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:40.060063 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:40.060066 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:40.060067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:40.060224 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:40.060682 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:00:40.060691 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:00:40.061009 (+ 318us) client_negotiation.cc:769] Sending connection context
0929 18:00:40.061201 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 18:00:40.061501 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":202,"threads_started":1}
I20260929 18:00:40.063997 22399 heartbeater.cc:347] Connected to a master server at 127.7.118.62:33763
I20260929 18:00:40.064316 22399 heartbeater.cc:464] Registering TS with master...
I20260929 18:00:40.065056 22399 heartbeater.cc:511] Master 127.7.118.62:33763 requested a full tablet report, sending...
I20260929 18:00:40.066421 21939 ts_manager.cc:194] Registered new tserver with Master: 596cf94ea8b44360adf9111ca5ba8485 (127.7.118.3:40605)
I20260929 18:00:40.067032 21939 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.3@KRBTEST.COM'} at 127.7.118.3:44289
I20260929 18:00:40.072032 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:00:40.077075 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:40.072831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32912 (local address 127.7.118.62:33763)
0929 18:00:40.073022 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:40.073027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:40.073187 (+ 160us) server_negotiation.cc:407] Connection header received
0929 18:00:40.073447 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:40.073449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:40.073563 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:40.073699 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:00:40.073968 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:40.074556 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:40.075483 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:40.075717 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:40.075842 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:40.076202 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:00:40.076640 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:00:40.076801 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:40.076835 (+ 34us) server_negotiation.cc:299] Negotiation successful
0929 18:00:40.076903 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 18:00:40.077966 21939 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:32912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:00:40.082142 21939 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:00:40.091713 22063 tablet_service.cc:1467] Processing CreateTablet for tablet b5849db6ffd844fcad9b41a1698b8945 (DEFAULT_TABLE table=test-table [id=0b9f08eb8448479ba40b7be8d49b846b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:00:40.092504 22063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5849db6ffd844fcad9b41a1698b8945. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:00:40.095715 22021 tablet_bootstrap.cc:492] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed: Bootstrap starting.
I20260929 18:00:40.095853 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:40.090545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:40605 (local address 127.0.0.1:43058)
0929 18:00:40.090712 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:40.090727 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:40.090809 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:40.091785 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:40.091789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:40.091807 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:00:40.091931 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:40.091938 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:40.093073 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:40.093078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:40.094318 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:40.094330 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:40.095328 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:40.095331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:40.095429 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:40.095481 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 18:00:40.095597 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 18:00:40.095698 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":1}
I20260929 18:00:40.096014 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:40.090850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43058 (local address 127.7.118.3:40605)
0929 18:00:40.091342 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:40.091353 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:40.091372 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:00:40.091439 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:40.091443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:40.091623 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:40.091815 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:00:40.092088 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:40.092946 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:40.094471 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:40.095182 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:40.095309 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:40.095473 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:40.095635 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 18:00:40.095738 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":145,"threads_started":1}
I20260929 18:00:40.096757 22021 tablet_bootstrap.cc:654] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed: Neither blocks nor log segments found. Creating new log.
I20260929 18:00:40.098451 22408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:40.090627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:44891 (local address 127.0.0.1:58176)
0929 18:00:40.091015 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:40.091027 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:40.091103 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:40.091851 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:40.091854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:40.091874 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:00:40.091990 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:40.091997 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:40.093515 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:40.093518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:40.094657 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:40.094666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:40.097869 (+ 3203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:40.097873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:40.098009 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:40.098052 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 18:00:40.098143 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 18:00:40.098289 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":1,"thread_start_us":125,"threads_started":1}
I20260929 18:00:40.098469 22021 tablet_bootstrap.cc:492] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed: No bootstrap required, opened a new log
I20260929 18:00:40.098487 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:40.090781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58176 (local address 127.7.118.2:44891)
0929 18:00:40.091383 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:40.091395 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:40.091417 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:00:40.091490 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:40.091496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:40.091683 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:40.092134 (+ 451us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:00:40.092306 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:40.093365 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:40.094794 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:40.097722 (+ 2928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:40.097846 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:40.098013 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:40.098191 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 18:00:40.098336 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":113,"threads_started":1}
I20260929 18:00:40.098572 22021 ts_tablet_manager.cc:1397] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:00:40.099139 22021 raft_consensus.cc:348] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } } peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } }
I20260929 18:00:40.099278 22021 raft_consensus.cc:374] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:00:40.099326 22021 raft_consensus.cc:729] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0d030cb1647419cbf29b0991405c7ed, State: Initialized, Role: FOLLOWER
I20260929 18:00:40.099454 22021 consensus_queue.cc:260] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } } peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } }
I20260929 18:00:40.099756 22021 ts_tablet_manager.cc:1428] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:00:40.099640 22333 tablet_service.cc:1467] Processing CreateTablet for tablet b5849db6ffd844fcad9b41a1698b8945 (DEFAULT_TABLE table=test-table [id=0b9f08eb8448479ba40b7be8d49b846b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:00:40.100926 22333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5849db6ffd844fcad9b41a1698b8945. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:00:40.102481 22201 tablet_service.cc:1467] Processing CreateTablet for tablet b5849db6ffd844fcad9b41a1698b8945 (DEFAULT_TABLE table=test-table [id=0b9f08eb8448479ba40b7be8d49b846b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:00:40.103710 22201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5849db6ffd844fcad9b41a1698b8945. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:00:40.110040 22412 tablet_bootstrap.cc:492] T b5849db6ffd844fcad9b41a1698b8945 P 596cf94ea8b44360adf9111ca5ba8485: Bootstrap starting.
I20260929 18:00:40.112416 22413 tablet_bootstrap.cc:492] T b5849db6ffd844fcad9b41a1698b8945 P b889ace27b9f4ee18982909cd1d47114: Bootstrap starting.
I20260929 18:00:40.113320 22412 tablet_bootstrap.cc:654] T b5849db6ffd844fcad9b41a1698b8945 P 596cf94ea8b44360adf9111ca5ba8485: Neither blocks nor log segments found. Creating new log.
I20260929 18:00:40.114782 22412 log.cc:824] T b5849db6ffd844fcad9b41a1698b8945 P 596cf94ea8b44360adf9111ca5ba8485: Log is configured to *not* fsync() on all Append() calls
I20260929 18:00:40.115480 22413 tablet_bootstrap.cc:654] T b5849db6ffd844fcad9b41a1698b8945 P b889ace27b9f4ee18982909cd1d47114: Neither blocks nor log segments found. Creating new log.
I20260929 18:00:40.116451 22413 log.cc:824] T b5849db6ffd844fcad9b41a1698b8945 P b889ace27b9f4ee18982909cd1d47114: Log is configured to *not* fsync() on all Append() calls
I20260929 18:00:40.117384 22412 tablet_bootstrap.cc:492] T b5849db6ffd844fcad9b41a1698b8945 P 596cf94ea8b44360adf9111ca5ba8485: No bootstrap required, opened a new log
I20260929 18:00:40.117681 22412 ts_tablet_manager.cc:1397] T b5849db6ffd844fcad9b41a1698b8945 P 596cf94ea8b44360adf9111ca5ba8485: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 18:00:40.118918 22413 tablet_bootstrap.cc:492] T b5849db6ffd844fcad9b41a1698b8945 P b889ace27b9f4ee18982909cd1d47114: No bootstrap required, opened a new log
I20260929 18:00:40.119138 22413 ts_tablet_manager.cc:1397] T b5849db6ffd844fcad9b41a1698b8945 P b889ace27b9f4ee18982909cd1d47114: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 18:00:40.120942 22412 raft_consensus.cc:348] T b5849db6ffd844fcad9b41a1698b8945 P 596cf94ea8b44360adf9111ca5ba8485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } } peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } }
I20260929 18:00:40.121199 22412 raft_consensus.cc:374] T b5849db6ffd844fcad9b41a1698b8945 P 596cf94ea8b44360adf9111ca5ba8485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:00:40.121280 22412 raft_consensus.cc:729] T b5849db6ffd844fcad9b41a1698b8945 P 596cf94ea8b44360adf9111ca5ba8485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596cf94ea8b44360adf9111ca5ba8485, State: Initialized, Role: FOLLOWER
I20260929 18:00:40.121785 22412 consensus_queue.cc:260] T b5849db6ffd844fcad9b41a1698b8945 P 596cf94ea8b44360adf9111ca5ba8485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } } peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } }
W20260929 18:00:40.122319 22130 tablet.cc:2369] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:00:40.122547 22399 heartbeater.cc:503] Master 127.7.118.62:33763 was elected leader, sending a full tablet report...
I20260929 18:00:40.122507 22413 raft_consensus.cc:348] T b5849db6ffd844fcad9b41a1698b8945 P b889ace27b9f4ee18982909cd1d47114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } } peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } }
I20260929 18:00:40.122800 22413 raft_consensus.cc:374] T b5849db6ffd844fcad9b41a1698b8945 P b889ace27b9f4ee18982909cd1d47114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:00:40.122879 22413 raft_consensus.cc:729] T b5849db6ffd844fcad9b41a1698b8945 P b889ace27b9f4ee18982909cd1d47114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b889ace27b9f4ee18982909cd1d47114, State: Initialized, Role: FOLLOWER
I20260929 18:00:40.123307 22413 consensus_queue.cc:260] T b5849db6ffd844fcad9b41a1698b8945 P b889ace27b9f4ee18982909cd1d47114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } } peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } }
I20260929 18:00:40.124091 22412 ts_tablet_manager.cc:1428] T b5849db6ffd844fcad9b41a1698b8945 P 596cf94ea8b44360adf9111ca5ba8485: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:00:40.124370 22413 ts_tablet_manager.cc:1428] T b5849db6ffd844fcad9b41a1698b8945 P b889ace27b9f4ee18982909cd1d47114: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:00:40.124410 22267 heartbeater.cc:503] Master 127.7.118.62:33763 was elected leader, sending a full tablet report...
I20260929 18:00:40.163446 22135 raft_consensus.cc:482] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:00:40.163806 22135 raft_consensus.cc:504] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } } peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } }
I20260929 18:00:40.165086 22135 leader_election.cc:288] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b889ace27b9f4ee18982909cd1d47114 (127.7.118.2:44891), 596cf94ea8b44360adf9111ca5ba8485 (127.7.118.3:40605)
I20260929 18:00:40.169325 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:40.165888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:35161 (local address 127.7.118.2:44891)
0929 18:00:40.166020 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:40.166025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:40.166039 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:00:40.166100 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:40.166106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:40.166166 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:40.166226 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:00:40.166609 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:40.167164 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:40.168268 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:40.168693 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:40.168817 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:40.168884 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:40.169104 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 18:00:40.169183 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:00:40.169328 22418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:40.165437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:44891 (local address 127.7.118.1:35161)
0929 18:00:40.165775 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:40.165792 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:40.165929 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:40.166307 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:40.166311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:40.166327 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:00:40.166492 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:40.166506 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:40.167285 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:40.167291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:40.168120 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:40.168129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:40.168828 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:40.168832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:40.168936 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:40.168992 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 18:00:40.169086 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 18:00:40.169174 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":88,"threads_started":1}
I20260929 18:00:40.170063 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:40.165832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:43961 (local address 127.7.118.3:40605)
0929 18:00:40.166003 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:40.166009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:40.166027 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:00:40.166257 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:40.166263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:40.166349 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:40.166443 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:00:40.166929 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:40.167483 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:40.168624 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:40.169257 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:40.169395 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:40.169460 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:40.169851 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 18:00:40.169935 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 18:00:40.170074 22419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:40.165437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:40605 (local address 127.7.118.1:43961)
0929 18:00:40.165923 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:40.165940 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:40.166043 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:40.166487 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:40.166490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:40.166572 (+ 82us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:00:40.166776 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:40.166785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:40.167607 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:40.167615 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:40.168472 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:40.168482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:40.169397 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:40.169401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:40.169522 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:40.169547 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:00:40.169831 (+ 284us) client_negotiation.cc:240] Negotiation successful
0929 18:00:40.169919 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":34,"thread_start_us":81,"threads_started":1}
I20260929 18:00:40.170332 22220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5849db6ffd844fcad9b41a1698b8945" candidate_uuid: "d0d030cb1647419cbf29b0991405c7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b889ace27b9f4ee18982909cd1d47114" is_pre_election: true
I20260929 18:00:40.170742 22220 raft_consensus.cc:2454] T b5849db6ffd844fcad9b41a1698b8945 P b889ace27b9f4ee18982909cd1d47114 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0d030cb1647419cbf29b0991405c7ed in term 0.
I20260929 18:00:40.170900 22353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5849db6ffd844fcad9b41a1698b8945" candidate_uuid: "d0d030cb1647419cbf29b0991405c7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596cf94ea8b44360adf9111ca5ba8485" is_pre_election: true
I20260929 18:00:40.171239 22353 raft_consensus.cc:2454] T b5849db6ffd844fcad9b41a1698b8945 P 596cf94ea8b44360adf9111ca5ba8485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0d030cb1647419cbf29b0991405c7ed in term 0.
I20260929 18:00:40.171288 22016 leader_election.cc:302] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b889ace27b9f4ee18982909cd1d47114, d0d030cb1647419cbf29b0991405c7ed; no voters:
I20260929 18:00:40.171660 22135 raft_consensus.cc:2781] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:00:40.171774 22135 raft_consensus.cc:482] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:00:40.171830 22135 raft_consensus.cc:3037] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:00:40.173630 22135 raft_consensus.cc:504] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } } peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } }
I20260929 18:00:40.175173 22135 leader_election.cc:288] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 1 election: Requested vote from peers b889ace27b9f4ee18982909cd1d47114 (127.7.118.2:44891), 596cf94ea8b44360adf9111ca5ba8485 (127.7.118.3:40605)
I20260929 18:00:40.175539 22220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5849db6ffd844fcad9b41a1698b8945" candidate_uuid: "d0d030cb1647419cbf29b0991405c7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b889ace27b9f4ee18982909cd1d47114"
I20260929 18:00:40.175716 22220 raft_consensus.cc:3037] T b5849db6ffd844fcad9b41a1698b8945 P b889ace27b9f4ee18982909cd1d47114 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:00:40.176287 22353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5849db6ffd844fcad9b41a1698b8945" candidate_uuid: "d0d030cb1647419cbf29b0991405c7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596cf94ea8b44360adf9111ca5ba8485"
I20260929 18:00:40.176430 22353 raft_consensus.cc:3037] T b5849db6ffd844fcad9b41a1698b8945 P 596cf94ea8b44360adf9111ca5ba8485 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:00:40.177579 22220 raft_consensus.cc:2454] T b5849db6ffd844fcad9b41a1698b8945 P b889ace27b9f4ee18982909cd1d47114 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0d030cb1647419cbf29b0991405c7ed in term 1.
I20260929 18:00:40.178042 22353 raft_consensus.cc:2454] T b5849db6ffd844fcad9b41a1698b8945 P 596cf94ea8b44360adf9111ca5ba8485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0d030cb1647419cbf29b0991405c7ed in term 1.
I20260929 18:00:40.178066 22016 leader_election.cc:302] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b889ace27b9f4ee18982909cd1d47114, d0d030cb1647419cbf29b0991405c7ed; no voters:
I20260929 18:00:40.178308 22135 raft_consensus.cc:2781] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:00:40.178536 22135 raft_consensus.cc:686] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 1 LEADER]: Becoming Leader. State: Replica: d0d030cb1647419cbf29b0991405c7ed, State: Running, Role: LEADER
I20260929 18:00:40.178875 22135 consensus_queue.cc:237] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } } peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } }
I20260929 18:00:40.182662 21938 catalog_manager.cc:5495] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to d0d030cb1647419cbf29b0991405c7ed (127.7.118.1). New cstate: current_term: 1 leader_uuid: "d0d030cb1647419cbf29b0991405c7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } health_report { overall_health: HEALTHY } } }
I20260929 18:00:40.205415 22422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:40.201117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55304 (local address 127.7.118.1:33205)
0929 18:00:40.201473 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:40.201480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:40.201494 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:00:40.201575 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:40.201578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:40.201634 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:40.201739 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:00:40.201995 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:40.202609 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:40.203509 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:40.203749 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:40.203860 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:40.204399 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:00:40.204904 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:00:40.205021 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:40.205141 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 18:00:40.205205 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":192,"threads_started":1}
I20260929 18:00:40.210103 22220 raft_consensus.cc:1261] T b5849db6ffd844fcad9b41a1698b8945 P b889ace27b9f4ee18982909cd1d47114 [term 1 FOLLOWER]: Refusing update from remote peer d0d030cb1647419cbf29b0991405c7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:00:40.210141 22353 raft_consensus.cc:1261] T b5849db6ffd844fcad9b41a1698b8945 P 596cf94ea8b44360adf9111ca5ba8485 [term 1 FOLLOWER]: Refusing update from remote peer d0d030cb1647419cbf29b0991405c7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:00:40.211014 22420 consensus_queue.cc:1035] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:00:40.211251 22135 consensus_queue.cc:1035] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:00:40.219432 22423 mvcc.cc:205] Tried to move back new op lower bound from 7334727025492529152 to 7334727025375911936. Current Snapshot: MvccSnapshot[applied={T|T < 7334727025492529152}]
I20260929 18:00:40.229550 21938 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:32912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:00:40.229765 21938 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:32912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:00:40.232699 21938 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627: Sending DeleteTablet for 3 replicas of tablet b5849db6ffd844fcad9b41a1698b8945
I20260929 18:00:40.233664 22333 tablet_service.cc:1514] Processing DeleteTablet for tablet b5849db6ffd844fcad9b41a1698b8945 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:00:40 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:43058
I20260929 18:00:40.233664 22201 tablet_service.cc:1514] Processing DeleteTablet for tablet b5849db6ffd844fcad9b41a1698b8945 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:00:40 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:58176
I20260929 18:00:40.234329 22433 tablet_replica.cc:330] T b5849db6ffd844fcad9b41a1698b8945 P 596cf94ea8b44360adf9111ca5ba8485: stopping tablet replica
I20260929 18:00:40.234508 22434 tablet_replica.cc:330] T b5849db6ffd844fcad9b41a1698b8945 P b889ace27b9f4ee18982909cd1d47114: stopping tablet replica
I20260929 18:00:40.234563 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 21979
I20260929 18:00:40.234887 22434 raft_consensus.cc:2229] T b5849db6ffd844fcad9b41a1698b8945 P b889ace27b9f4ee18982909cd1d47114 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:00:40.234937 22433 raft_consensus.cc:2229] T b5849db6ffd844fcad9b41a1698b8945 P 596cf94ea8b44360adf9111ca5ba8485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:00:40.235329 22434 raft_consensus.cc:2258] T b5849db6ffd844fcad9b41a1698b8945 P b889ace27b9f4ee18982909cd1d47114 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:00:40.235489 22433 raft_consensus.cc:2258] T b5849db6ffd844fcad9b41a1698b8945 P 596cf94ea8b44360adf9111ca5ba8485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:00:40.236898 22433 ts_tablet_manager.cc:1905] T b5849db6ffd844fcad9b41a1698b8945 P 596cf94ea8b44360adf9111ca5ba8485: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:00:40.237090 22434 ts_tablet_manager.cc:1905] T b5849db6ffd844fcad9b41a1698b8945 P b889ace27b9f4ee18982909cd1d47114: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:00:40.239351 22433 ts_tablet_manager.cc:1918] T b5849db6ffd844fcad9b41a1698b8945 P 596cf94ea8b44360adf9111ca5ba8485: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:00:40.239485 22433 log.cc:1196] T b5849db6ffd844fcad9b41a1698b8945 P 596cf94ea8b44360adf9111ca5ba8485: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/wal/wals/b5849db6ffd844fcad9b41a1698b8945
I20260929 18:00:40.239842 22433 ts_tablet_manager.cc:1939] T b5849db6ffd844fcad9b41a1698b8945 P 596cf94ea8b44360adf9111ca5ba8485: Deleting consensus metadata
I20260929 18:00:40.240595 22434 ts_tablet_manager.cc:1918] T b5849db6ffd844fcad9b41a1698b8945 P b889ace27b9f4ee18982909cd1d47114: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:00:40.240720 22434 log.cc:1196] T b5849db6ffd844fcad9b41a1698b8945 P b889ace27b9f4ee18982909cd1d47114: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/wal/wals/b5849db6ffd844fcad9b41a1698b8945
I20260929 18:00:40.241063 22434 ts_tablet_manager.cc:1939] T b5849db6ffd844fcad9b41a1698b8945 P b889ace27b9f4ee18982909cd1d47114: Deleting consensus metadata
I20260929 18:00:40.241159 21928 catalog_manager.cc:4841] TS 596cf94ea8b44360adf9111ca5ba8485 (127.7.118.3:40605): tablet b5849db6ffd844fcad9b41a1698b8945 (table test-table [id=0b9f08eb8448479ba40b7be8d49b846b]) successfully deleted
I20260929 18:00:40.242475 21926 catalog_manager.cc:4841] TS b889ace27b9f4ee18982909cd1d47114 (127.7.118.2:44891): tablet b5849db6ffd844fcad9b41a1698b8945 (table test-table [id=0b9f08eb8448479ba40b7be8d49b846b]) successfully deleted
W20260929 18:00:40.243764 21926 connection.cc:535] client connection to 127.7.118.1:33205 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 18:00:40.243942 21926 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 18:00:40.244021 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 22134
I20260929 18:00:40.245303 22408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:40.245033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:33205 (local address 127.0.0.1:55306)
0929 18:00:40.245147 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:40.245215 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.1:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 18:00:40.245518 21926 catalog_manager.cc:4584] TS d0d030cb1647419cbf29b0991405c7ed (127.7.118.1:33205): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b5849db6ffd844fcad9b41a1698b8945: Network error: Client connection negotiation failed: client connection to 127.7.118.1:33205: connect: Connection refused (error 111)
I20260929 18:00:40.250111 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 22270
I20260929 18:00:40.256100 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 21908
I20260929 18:00:40.263106 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:33763
--webserver_interface=127.7.118.62
--webserver_port=39109
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:33763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:00:40.382599 22436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:40.382974 22436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:40.383070 22436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:00:40.387571 22436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:00:40.387692 22436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:40.387728 22436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:40.387768 22436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:00:40.387835 22436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:00:40.393844 22436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:33763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:33763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=39109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:40.395172 22436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:40.396462 22436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:00:40.406420 22441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:40.406420 22444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:40.406484 22436 server_base.cc:1034] running on GCE node
W20260929 18:00:40.406440 22442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:40.407356 22436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:40.408710 22436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:40.409955 22436 hybrid_clock.cc:648] HybridClock initialized: now 1790704840409913 us; error 63 us; skew 500 ppm
Sep 29 18:00:40 dist-test-slave-r7ph krb5kdc[21360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704840, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:40.413496 22436 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 18:00:40.414727 22436 webserver.cc:460] Webserver started at http://127.7.118.62:39109/ using document root <none> and password file <none>
I20260929 18:00:40.415398 22436 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:40.415510 22436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:00:40.420233 22436 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:00:40.423113 22451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:40.424707 22436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:00:40.424880 22436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "8a24a1dd183f432da92e1a3f81726627"
format_stamp: "Formatted at 2026-09-29 18:00:37 on dist-test-slave-r7ph"
server_key: "3e5f2ab3a17cbd435f56dd7f3c28236d"
server_key_iv: "9c208ea10dd20b632694d814569798ce"
server_key_version: "encryptionkey@0"
I20260929 18:00:40.425551 22436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:40.447912 22436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:40.451601 22436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:40.451910 22436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:40.459905 22436 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:33763
I20260929 18:00:40.460201 22503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:33763 every 8 connection(s)
I20260929 18:00:40.461185 22436 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:00:40.465977 22504 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:00:40.467998 22504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627: Bootstrap starting.
I20260929 18:00:40.471014 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 22436
I20260929 18:00:40.471545 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:33205
--local_ip_for_outbound_sockets=127.7.118.1
--tserver_master_addrs=127.7.118.62:33763
--webserver_port=38141
--webserver_interface=127.7.118.1
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:00:40.487295 22504 log.cc:824] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627: Log is configured to *not* fsync() on all Append() calls
I20260929 18:00:40.499774 22504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:00:40.500232 22504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627: Bootstrap complete.
I20260929 18:00:40.504670 22504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a24a1dd183f432da92e1a3f81726627" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33763 } }
I20260929 18:00:40.505165 22504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a24a1dd183f432da92e1a3f81726627, State: Initialized, Role: FOLLOWER
I20260929 18:00:40.505724 22504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a24a1dd183f432da92e1a3f81726627" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33763 } }
I20260929 18:00:40.505899 22504 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:00:40.505999 22504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:00:40.506138 22504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [term 2 FOLLOWER]: Advancing to term 3
I20260929 18:00:40.508320 22504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a24a1dd183f432da92e1a3f81726627" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33763 } }
I20260929 18:00:40.508706 22504 leader_election.cc:302] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a24a1dd183f432da92e1a3f81726627; no voters:
I20260929 18:00:40.509088 22504 leader_election.cc:288] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 18:00:40.509181 22508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 18:00:40.509431 22508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [term 3 LEADER]: Becoming Leader. State: Replica: 8a24a1dd183f432da92e1a3f81726627, State: Running, Role: LEADER
I20260929 18:00:40.509828 22508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a24a1dd183f432da92e1a3f81726627" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33763 } }
I20260929 18:00:40.510530 22504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:00:40.511353 22509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8a24a1dd183f432da92e1a3f81726627" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a24a1dd183f432da92e1a3f81726627" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33763 } } }
I20260929 18:00:40.511499 22509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [sys.catalog]: This master's current role is: LEADER
I20260929 18:00:40.512288 22510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a24a1dd183f432da92e1a3f81726627. Latest consensus state: current_term: 3 leader_uuid: "8a24a1dd183f432da92e1a3f81726627" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a24a1dd183f432da92e1a3f81726627" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33763 } } }
I20260929 18:00:40.512362 22515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:00:40.512454 22510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a24a1dd183f432da92e1a3f81726627 [sys.catalog]: This master's current role is: LEADER
I20260929 18:00:40.518082 22515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:00:40.519109 22515 catalog_manager.cc:1259] Loaded cluster ID: bb83e00edd5f49829d3c0e9d1eada7cc
I20260929 18:00:40.519207 22515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 18:00:40.520289 22515 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1f2af6f9ad google::LogMessage::Fail() at ??:0
@ 0x7f1f2af7288a google::LogMessage::SendToLog() at ??:0
@ 0x7f1f2af6f4de google::LogMessage::Flush() at ??:0
@ 0x7f1f2af70a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1f358aefdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f1f358afd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1f358ac579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f1f358f0640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1f2d1c13d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1f2d1c1c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f1f2d1c33f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1f2d1b2512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1f2e7f66db start_thread at ??:0
@ 0x7f1f29a6d61f clone at ??:0
W20260929 18:00:40.633502 22506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:40.634361 22506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:40.634645 22506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:00:40.640105 22506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:40.640197 22506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:40.640265 22506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 18:00:40.646361 22506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:33205
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=38141
--tserver_master_addrs=127.7.118.62:33763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:40.647608 22506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:40.648744 22506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:00:40.660127 22546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:40.660127 22545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:40.660132 22548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:40.660704 22506 server_base.cc:1034] running on GCE node
I20260929 18:00:40.661332 22506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:40.662130 22506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:40.663621 22506 hybrid_clock.cc:648] HybridClock initialized: now 1790704840663587 us; error 55 us; skew 500 ppm
Sep 29 18:00:40 dist-test-slave-r7ph krb5kdc[21360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704840, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:40.667619 22506 init.cc:377] Logged in from keytab as kudu/127.7.118.1@KRBTEST.COM (short username kudu)
I20260929 18:00:40.668926 22506 webserver.cc:460] Webserver started at http://127.7.118.1:38141/ using document root <none> and password file <none>
I20260929 18:00:40.669615 22506 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:40.669690 22506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:00:40.674736 22506 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:00:40.677796 22555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:40.679792 22506 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:00:40.680007 22506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "d0d030cb1647419cbf29b0991405c7ed"
format_stamp: "Formatted at 2026-09-29 18:00:38 on dist-test-slave-r7ph"
server_key: "2d221812c87fbc319150cd86b28ad975"
server_key_iv: "3338f886ffdca4aa7f6513fb8a4716fa"
server_key_version: "encryptionkey@0"
I20260929 18:00:40.680719 22506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:40.710170 22506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:40.714607 22506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:40.714859 22506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:40.715571 22506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:00:40.716902 22562 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:00:40.722931 22506 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:00:40.723067 22506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260929 18:00:40.723166 22506 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:00:40.725340 22506 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:00:40.725435 22506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 18:00:40.725783 22562 tablet_bootstrap.cc:492] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed: Bootstrap starting.
I20260929 18:00:40.737092 22506 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:33205
I20260929 18:00:40.737110 22669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:33205 every 8 connection(s)
I20260929 18:00:40.738446 22506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 18:00:40.740442 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 22506
I20260929 18:00:40.740903 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:44891
--local_ip_for_outbound_sockets=127.7.118.2
--tserver_master_addrs=127.7.118.62:33763
--webserver_port=42555
--webserver_interface=127.7.118.2
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:00:40.751188 22562 log.cc:824] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed: Log is configured to *not* fsync() on all Append() calls
I20260929 18:00:40.761720 22562 tablet_bootstrap.cc:492] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:00:40.762188 22562 tablet_bootstrap.cc:492] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed: Bootstrap complete.
I20260929 18:00:40.763752 22562 ts_tablet_manager.cc:1397] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed: Time spent bootstrapping tablet: real 0.038s user 0.033s sys 0.004s
I20260929 18:00:40.768721 22562 raft_consensus.cc:348] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } } peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } }
I20260929 18:00:40.769371 22562 raft_consensus.cc:729] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0d030cb1647419cbf29b0991405c7ed, State: Initialized, Role: FOLLOWER
I20260929 18:00:40.769972 22562 consensus_queue.cc:260] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } } peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } }
I20260929 18:00:40.771162 22562 ts_tablet_manager.cc:1428] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
W20260929 18:00:40.772255 22672 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:00:40.740953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.1:43627)
0929 18:00:40.742581 (+ 1628us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:40.742698 (+ 117us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:40.743566 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:40.771436 (+ 27870us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1336,"thread_start_us":189,"threads_started":1}
W20260929 18:00:40.772778 22670 heartbeater.cc:644] Failed to heartbeat to 127.7.118.62:33763 (0 consecutive failures): Network error: Failed to ping master at 127.7.118.62:33763: Client connection negotiation failed: client connection to 127.7.118.62:33763: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 18:00:40.773643 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:40.773322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.1:38357)
0929 18:00:40.773507 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:40.773578 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260929 18:00:40.774482 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:40.774279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.1:55427)
0929 18:00:40.774385 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:40.774429 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 18:00:40.774725 22670 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:00:40.873736 22673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:40.874068 22673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:40.874115 22673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:00:40.878685 22673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:40.878768 22673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:40.878835 22673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 18:00:40.884732 22673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:44891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=42555
--tserver_master_addrs=127.7.118.62:33763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:40.886010 22673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:40.887217 22673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:00:40.898056 22681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:40.898064 22680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:40.898064 22683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:40.898733 22673 server_base.cc:1034] running on GCE node
I20260929 18:00:40.899307 22673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:40.900106 22673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:40.901316 22673 hybrid_clock.cc:648] HybridClock initialized: now 1790704840901295 us; error 38 us; skew 500 ppm
Sep 29 18:00:40 dist-test-slave-r7ph krb5kdc[21360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704840, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:40.904548 22673 init.cc:377] Logged in from keytab as kudu/127.7.118.2@KRBTEST.COM (short username kudu)
I20260929 18:00:40.905742 22673 webserver.cc:460] Webserver started at http://127.7.118.2:42555/ using document root <none> and password file <none>
I20260929 18:00:40.906458 22673 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:40.906563 22673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:00:40.911227 22673 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 18:00:40.913730 22690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:40.915336 22673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:00:40.915535 22673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "b889ace27b9f4ee18982909cd1d47114"
format_stamp: "Formatted at 2026-09-29 18:00:38 on dist-test-slave-r7ph"
server_key: "c6e2f7f6df0473c313c2cd4fa86f1063"
server_key_iv: "9af37916a4a1546e2e24f16376f62967"
server_key_version: "encryptionkey@0"
I20260929 18:00:40.916210 22673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:40.937376 22673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:40.941392 22673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:40.941635 22673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:40.942409 22673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:00:40.943342 22673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:00:40.943421 22673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:40.943495 22673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:00:40.943549 22673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:40.953589 22673 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:44891
I20260929 18:00:40.953613 22803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:44891 every 8 connection(s)
I20260929 18:00:40.954844 22673 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 18:00:40.959633 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:40.957492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.2:54401)
0929 18:00:40.958106 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:40.958237 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":147,"threads_started":1}
I20260929 18:00:40.960235 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 22673
W20260929 18:00:40.960333 22804 heartbeater.cc:644] Failed to heartbeat to 127.7.118.62:33763 (0 consecutive failures): Network error: Failed to ping master at 127.7.118.62:33763: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
I20260929 18:00:40.960779 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:40605
--local_ip_for_outbound_sockets=127.7.118.3
--tserver_master_addrs=127.7.118.62:33763
--webserver_port=35515
--webserver_interface=127.7.118.3
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:00:40.961174 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:40.960813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.2:56453)
0929 18:00:40.961021 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:40.961084 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 18:00:40.962019 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:40.961793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.2:38079)
0929 18:00:40.961912 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:40.961960 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 18:00:40.962366 22804 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:00:41.089435 22807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:41.089808 22807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:41.089898 22807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:00:41.094488 22807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:41.094581 22807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:41.094676 22807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 18:00:41.100476 22807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:40605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=35515
--tserver_master_addrs=127.7.118.62:33763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:41.101671 22807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:41.102890 22807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:00:41.114347 22815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:41.114456 22807 server_base.cc:1034] running on GCE node
W20260929 18:00:41.114347 22812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:41.114756 22813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:41.115235 22807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:41.116029 22807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:41.117264 22807 hybrid_clock.cc:648] HybridClock initialized: now 1790704841117235 us; error 45 us; skew 500 ppm
Sep 29 18:00:41 dist-test-slave-r7ph krb5kdc[21360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704841, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:41.120501 22807 init.cc:377] Logged in from keytab as kudu/127.7.118.3@KRBTEST.COM (short username kudu)
I20260929 18:00:41.121497 22807 webserver.cc:460] Webserver started at http://127.7.118.3:35515/ using document root <none> and password file <none>
I20260929 18:00:41.122223 22807 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:41.122346 22807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:00:41.127333 22807 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 18:00:41.129838 22822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:41.131306 22807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:00:41.131438 22807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "596cf94ea8b44360adf9111ca5ba8485"
format_stamp: "Formatted at 2026-09-29 18:00:38 on dist-test-slave-r7ph"
server_key: "4403b0b34a242423303ec92dd8bd9ed1"
server_key_iv: "b61535a0707a6455a9b16118ea08af1e"
server_key_version: "encryptionkey@0"
I20260929 18:00:41.132143 22807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:41.144286 22807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:41.148097 22807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:41.148275 22807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:41.148983 22807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:00:41.149893 22807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:00:41.149941 22807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:41.149983 22807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:00:41.149999 22807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:41.160567 22807 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:40605
I20260929 18:00:41.160607 22935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:40605 every 8 connection(s)
I20260929 18:00:41.161744 22807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 18:00:41.166613 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:41.164270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.3:45017)
0929 18:00:41.164876 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:41.165003 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":145,"threads_started":1}
W20260929 18:00:41.167296 22936 heartbeater.cc:644] Failed to heartbeat to 127.7.118.62:33763 (0 consecutive failures): Network error: Failed to ping master at 127.7.118.62:33763: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
I20260929 18:00:41.168170 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:41.167870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.3:57121)
0929 18:00:41.168028 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:41.168097 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 18:00:41.169096 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:41.168745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.3:33795)
0929 18:00:41.168898 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:41.168974 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260929 18:00:41.169343 22936 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 18:00:41.169360 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 22807
I20260929 18:00:41.775980 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:41.775442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.1:50237)
0929 18:00:41.775765 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:41.775852 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":121,"threads_started":1}
I20260929 18:00:41.963637 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:41.963074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.2:38795)
0929 18:00:41.963431 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:41.963505 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":89,"threads_started":1}
I20260929 18:00:42.170693 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:42.170076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.3:41605)
0929 18:00:42.170501 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:42.170567 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":106,"threads_started":1}
I20260929 18:00:42.356099 22943 raft_consensus.cc:482] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:00:42.356387 22943 raft_consensus.cc:504] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } } peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } }
I20260929 18:00:42.357774 22943 leader_election.cc:288] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b889ace27b9f4ee18982909cd1d47114 (127.7.118.2:44891), 596cf94ea8b44360adf9111ca5ba8485 (127.7.118.3:40605)
Sep 29 18:00:42 dist-test-slave-r7ph krb5kdc[21360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704840, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for kudu/127.7.118.3@KRBTEST.COM
Sep 29 18:00:42 dist-test-slave-r7ph krb5kdc[21360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704840, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for kudu/127.7.118.2@KRBTEST.COM
I20260929 18:00:42.375085 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:42.358405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:60925 (local address 127.7.118.3:40605)
0929 18:00:42.358773 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:42.358800 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:42.358867 (+ 67us) server_negotiation.cc:407] Connection header received
0929 18:00:42.359880 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:42.359890 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:42.360406 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:42.360782 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:42.362145 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:42.363275 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:42.364867 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:42.365462 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:42.365669 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:42.369087 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:42.369096 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:42.369119 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:42.369168 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:42.371738 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:42.372603 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:42.372615 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:42.372619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:42.372778 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:42.373199 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:42.373203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:42.373205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:42.373726 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:42.373839 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:42.374568 (+ 729us) server_negotiation.cc:299] Negotiation successful
0929 18:00:42.374875 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":65,"threads_started":1}
I20260929 18:00:42.375070 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:42.358092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:44891 (local address 127.7.118.1:52547)
0929 18:00:42.358982 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:42.358995 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:42.359080 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:42.361321 (+ 2241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:42.361324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:42.361514 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:00:42.361948 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:42.361958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:42.363545 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:42.363548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:42.364552 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:42.364564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:42.365535 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:42.365543 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:42.365775 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:42.366690 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:00:42.366709 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:00:42.368944 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:00:42.371978 (+ 3034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:42.371980 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:42.371982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:42.372463 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:42.373104 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:42.373106 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:42.373107 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:42.373227 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:42.374180 (+ 953us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:00:42.374183 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:00:42.374330 (+ 147us) client_negotiation.cc:769] Sending connection context
0929 18:00:42.374453 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 18:00:42.374886 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"mutex_wait_us":665,"spinlock_wait_cycles":28288,"thread_start_us":98,"threads_started":1}
I20260929 18:00:42.375169 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:42.358612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:52547 (local address 127.7.118.2:44891)
0929 18:00:42.359086 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:42.359108 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:42.359148 (+ 40us) server_negotiation.cc:407] Connection header received
0929 18:00:42.360013 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:42.360020 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:42.360405 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:42.360772 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:42.362144 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:42.363392 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:42.364745 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:42.365366 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:42.365515 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:42.369220 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:42.369226 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:42.369240 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:42.369276 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:42.371793 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:42.372626 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:42.372637 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:42.372642 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:42.372782 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:42.373375 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:42.373378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:42.373380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:42.374055 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:42.374203 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:42.374635 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 18:00:42.374962 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":121,"threads_started":1}
I20260929 18:00:42.375177 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:42.358092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:40605 (local address 127.7.118.1:60925)
0929 18:00:42.358446 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:42.358462 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:42.358585 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:42.360912 (+ 2327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:42.360918 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:42.361331 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:00:42.361923 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:42.361941 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:42.363466 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:42.363472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:42.364556 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:42.364568 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:42.365621 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:42.365626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:42.365773 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:42.366708 (+ 935us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:00:42.366723 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:00:42.368631 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:00:42.371929 (+ 3298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:42.371938 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:42.371958 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:42.372454 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:42.372895 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:42.372902 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:42.372907 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:42.373085 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:42.373866 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:00:42.373872 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:00:42.374155 (+ 283us) client_negotiation.cc:769] Sending connection context
0929 18:00:42.374399 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 18:00:42.374975 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":394,"thread_start_us":73,"threads_started":1}
W20260929 18:00:42.377976 22557 leader_election.cc:341] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b889ace27b9f4ee18982909cd1d47114 (127.7.118.2:44891): Not found: Tablet not found: b5849db6ffd844fcad9b41a1698b8945
W20260929 18:00:42.378166 22559 leader_election.cc:341] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 596cf94ea8b44360adf9111ca5ba8485 (127.7.118.3:40605): Not found: Tablet not found: b5849db6ffd844fcad9b41a1698b8945
I20260929 18:00:42.378237 22559 leader_election.cc:302] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0d030cb1647419cbf29b0991405c7ed; no voters: 596cf94ea8b44360adf9111ca5ba8485, b889ace27b9f4ee18982909cd1d47114
I20260929 18:00:42.378564 22943 raft_consensus.cc:2726] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:00:42.777330 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:42.776932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.1:37869)
0929 18:00:42.777152 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:42.777234 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 18:00:42.965130 22948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:42.964605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.2:34127)
0929 18:00:42.964943 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:42.965021 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":82,"threads_started":1}
I20260929 18:00:43.172549 22949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:43.171891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.3:45133)
0929 18:00:43.172373 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:43.172445 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":88,"threads_started":1}
I20260929 18:00:43.779147 22950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:43.778651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.1:36783)
0929 18:00:43.778971 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:43.779040 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":89,"threads_started":1}
I20260929 18:00:43.966797 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:43.966167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.2:48873)
0929 18:00:43.966577 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:43.966669 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":151,"threads_started":1}
I20260929 18:00:44.039938 22952 raft_consensus.cc:482] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:00:44.040117 22952 raft_consensus.cc:504] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } } peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } }
I20260929 18:00:44.040635 22952 leader_election.cc:288] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b889ace27b9f4ee18982909cd1d47114 (127.7.118.2:44891), 596cf94ea8b44360adf9111ca5ba8485 (127.7.118.3:40605)
W20260929 18:00:44.041592 22559 leader_election.cc:341] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 596cf94ea8b44360adf9111ca5ba8485 (127.7.118.3:40605): Not found: Tablet not found: b5849db6ffd844fcad9b41a1698b8945
W20260929 18:00:44.041733 22557 leader_election.cc:341] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b889ace27b9f4ee18982909cd1d47114 (127.7.118.2:44891): Not found: Tablet not found: b5849db6ffd844fcad9b41a1698b8945
I20260929 18:00:44.041855 22557 leader_election.cc:302] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0d030cb1647419cbf29b0991405c7ed; no voters: 596cf94ea8b44360adf9111ca5ba8485, b889ace27b9f4ee18982909cd1d47114
I20260929 18:00:44.042044 22952 raft_consensus.cc:2726] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:00:44.174448 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:44.173650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.3:60383)
0929 18:00:44.174049 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:44.174114 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":114,"threads_started":1}
I20260929 18:00:44.780542 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:44.780123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.1:49965)
0929 18:00:44.780387 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:44.780447 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":77,"threads_started":1}
I20260929 18:00:44.968286 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:44.967758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.2:35415)
0929 18:00:44.968107 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:44.968182 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":98,"threads_started":1}
I20260929 18:00:45.175802 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:45.175335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.3:40705)
0929 18:00:45.175644 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:45.175705 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":92,"threads_started":1}
I20260929 18:00:45.782143 22957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:45.781683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.1:60467)
0929 18:00:45.781983 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:45.782043 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":74,"threads_started":1}
I20260929 18:00:45.969816 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:45.969328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.2:51263)
0929 18:00:45.969662 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:45.969721 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":81,"threads_started":1}
I20260929 18:00:46.177250 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:46.176727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.3:54171)
0929 18:00:46.177078 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:46.177150 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":85,"threads_started":1}
I20260929 18:00:46.783572 22960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:46.783079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.1:42827)
0929 18:00:46.783365 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:46.783430 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":85,"threads_started":1}
I20260929 18:00:46.971536 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:46.971000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.2:33723)
0929 18:00:46.971342 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:46.971418 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":88,"threads_started":1}
I20260929 18:00:47.030676 22962 raft_consensus.cc:482] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:00:47.030833 22962 raft_consensus.cc:504] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } } peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } }
I20260929 18:00:47.031284 22962 leader_election.cc:288] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b889ace27b9f4ee18982909cd1d47114 (127.7.118.2:44891), 596cf94ea8b44360adf9111ca5ba8485 (127.7.118.3:40605)
W20260929 18:00:47.032255 22559 leader_election.cc:341] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 596cf94ea8b44360adf9111ca5ba8485 (127.7.118.3:40605): Not found: Tablet not found: b5849db6ffd844fcad9b41a1698b8945
W20260929 18:00:47.032539 22557 leader_election.cc:341] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b889ace27b9f4ee18982909cd1d47114 (127.7.118.2:44891): Not found: Tablet not found: b5849db6ffd844fcad9b41a1698b8945
I20260929 18:00:47.032742 22557 leader_election.cc:302] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0d030cb1647419cbf29b0991405c7ed; no voters: 596cf94ea8b44360adf9111ca5ba8485, b889ace27b9f4ee18982909cd1d47114
I20260929 18:00:47.033025 22962 raft_consensus.cc:2726] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:00:47.179091 22963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:47.178400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.3:36173)
0929 18:00:47.178861 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:47.178960 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":157,"threads_started":1}
I20260929 18:00:47.785173 22964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:47.784544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.1:51403)
0929 18:00:47.784998 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:47.785066 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":87,"threads_started":1}
I20260929 18:00:47.973145 22965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:47.972559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.2:56505)
0929 18:00:47.972969 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:47.973041 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":84,"threads_started":1}
I20260929 18:00:48.180578 22966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:48.180058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.3:43743)
0929 18:00:48.180407 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:48.180477 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":77,"threads_started":1}
I20260929 18:00:48.786657 22967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:48.786157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.1:33223)
0929 18:00:48.786490 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:48.786556 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":135,"threads_started":1}
I20260929 18:00:48.813714 22968 raft_consensus.cc:482] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:00:48.813884 22968 raft_consensus.cc:504] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } } peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } }
I20260929 18:00:48.814483 22968 leader_election.cc:288] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b889ace27b9f4ee18982909cd1d47114 (127.7.118.2:44891), 596cf94ea8b44360adf9111ca5ba8485 (127.7.118.3:40605)
W20260929 18:00:48.815444 22557 leader_election.cc:341] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b889ace27b9f4ee18982909cd1d47114 (127.7.118.2:44891): Not found: Tablet not found: b5849db6ffd844fcad9b41a1698b8945
W20260929 18:00:48.815572 22559 leader_election.cc:341] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 596cf94ea8b44360adf9111ca5ba8485 (127.7.118.3:40605): Not found: Tablet not found: b5849db6ffd844fcad9b41a1698b8945
I20260929 18:00:48.815667 22559 leader_election.cc:302] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0d030cb1647419cbf29b0991405c7ed; no voters: 596cf94ea8b44360adf9111ca5ba8485, b889ace27b9f4ee18982909cd1d47114
I20260929 18:00:48.815838 22968 raft_consensus.cc:2726] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:00:48.974601 22969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:48.974089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.2:60933)
0929 18:00:48.974431 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:48.974500 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":78,"threads_started":1}
I20260929 18:00:49.182161 22970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:49.181629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.3:39085)
0929 18:00:49.181973 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:49.182062 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":101,"threads_started":1}
I20260929 18:00:49.788071 22971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:49.787565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.1:38039)
0929 18:00:49.787900 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:49.787970 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":98,"threads_started":1}
I20260929 18:00:49.976207 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:49.975627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.2:53723)
0929 18:00:49.976009 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:49.976092 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":108,"threads_started":1}
I20260929 18:00:50.183586 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:50.183126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.3:44285)
0929 18:00:50.183410 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:50.183484 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":77,"threads_started":1}
I20260929 18:00:50.789544 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:50.789090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.1:45513)
0929 18:00:50.789388 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:50.789448 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":82,"threads_started":1}
I20260929 18:00:50.978099 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:50.977487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.2:50017)
0929 18:00:50.977828 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:50.977895 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":91,"threads_started":1}
I20260929 18:00:51.185253 22976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:51.184630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.3:42473)
0929 18:00:51.185066 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:51.185147 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":117,"threads_started":1}
I20260929 18:00:51.791118 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:51.790624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.1:44653)
0929 18:00:51.790947 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:51.791014 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":99,"threads_started":1}
I20260929 18:00:51.979715 22978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:51.979162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.2:40109)
0929 18:00:51.979512 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:51.979583 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":97,"threads_started":1}
I20260929 18:00:52.186726 22979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:52.186200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.3:49343)
0929 18:00:52.186563 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:52.186625 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":160,"threads_started":1}
I20260929 18:00:52.792521 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:52.792032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.1:35043)
0929 18:00:52.792336 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:52.792425 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":90,"threads_started":1}
I20260929 18:00:52.981325 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:52.980782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.2:50893)
0929 18:00:52.981137 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:52.981208 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":98,"threads_started":1}
I20260929 18:00:53.188203 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:53.187743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.3:42991)
0929 18:00:53.188046 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:53.188107 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":113,"threads_started":1}
I20260929 18:00:53.793895 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:53.793418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.1:37031)
0929 18:00:53.793725 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:53.793793 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":81,"threads_started":1}
I20260929 18:00:53.982930 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:53.982476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.2:53819)
0929 18:00:53.982753 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:53.982831 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":112,"threads_started":1}
I20260929 18:00:54.189718 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:54.189237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.3:44617)
0929 18:00:54.189534 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:54.189608 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":90,"threads_started":1}
I20260929 18:00:54.245165 22986 raft_consensus.cc:482] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:00:54.245314 22986 raft_consensus.cc:504] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b889ace27b9f4ee18982909cd1d47114" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 44891 } } peers { permanent_uuid: "596cf94ea8b44360adf9111ca5ba8485" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 40605 } } peers { permanent_uuid: "d0d030cb1647419cbf29b0991405c7ed" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33205 } }
I20260929 18:00:54.245792 22986 leader_election.cc:288] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b889ace27b9f4ee18982909cd1d47114 (127.7.118.2:44891), 596cf94ea8b44360adf9111ca5ba8485 (127.7.118.3:40605)
W20260929 18:00:54.246896 22557 leader_election.cc:341] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b889ace27b9f4ee18982909cd1d47114 (127.7.118.2:44891): Not found: Tablet not found: b5849db6ffd844fcad9b41a1698b8945
W20260929 18:00:54.247012 22559 leader_election.cc:341] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 596cf94ea8b44360adf9111ca5ba8485 (127.7.118.3:40605): Not found: Tablet not found: b5849db6ffd844fcad9b41a1698b8945
I20260929 18:00:54.247134 22559 leader_election.cc:302] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0d030cb1647419cbf29b0991405c7ed; no voters: 596cf94ea8b44360adf9111ca5ba8485, b889ace27b9f4ee18982909cd1d47114
I20260929 18:00:54.247337 22986 raft_consensus.cc:2726] T b5849db6ffd844fcad9b41a1698b8945 P d0d030cb1647419cbf29b0991405c7ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:00:54.795622 22987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:54.795050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.1:60209)
0929 18:00:54.795433 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:54.795499 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":107,"threads_started":1}
I20260929 18:00:54.984555 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:54.983977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.2:53007)
0929 18:00:54.984378 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:54.984450 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":89,"threads_started":1}
I20260929 18:00:55.191341 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:55.190846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.3:40977)
0929 18:00:55.191181 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:55.191245 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":94,"threads_started":1}
I20260929 18:00:55.797315 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:55.796796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.1:60803)
0929 18:00:55.797132 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:55.797217 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":80,"threads_started":1}
I20260929 18:00:55.986434 22991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:55.985798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33763 (local address 127.7.118.2:35723)
0929 18:00:55.986124 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:55.986228 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":77,"threads_started":1}
I20260929 18:00:56.171269 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 22506
I20260929 18:00:56.177277 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 22673
I20260929 18:00:56.182855 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 22807
2026-09-29T18:00:56Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19808 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:00:56 dist-test-slave-r7ph krb5kdc[22994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:00:56 dist-test-slave-r7ph krb5kdc[22994](info): set up 2 sockets
Sep 29 18:00:56 dist-test-slave-r7ph krb5kdc[22994](info): commencing operation
krb5kdc: starting...
W20260929 18:00:57.266736 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 18:00:57 dist-test-slave-r7ph krb5kdc[22994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:00:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:00:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 18:00:57.402832 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:45259
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:42109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:45259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:00:57.522210 23007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:57.522675 23007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:57.522789 23007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:00:57.527197 23007 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 18:00:57.527299 23007 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 18:00:57.527325 23007 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 18:00:57.527362 23007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:00:57.527436 23007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:57.527494 23007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:57.527537 23007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:00:57.527572 23007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:00:57.533448 23007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:42109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:45259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:45259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:57.534756 23007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:57.535908 23007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:00:57.545897 23007 server_base.cc:1034] running on GCE node
W20260929 18:00:57.545917 23015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:57.545918 23013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:57.546238 23012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:57.546964 23007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:57.548198 23007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:57.549465 23007 hybrid_clock.cc:648] HybridClock initialized: now 1790704857549448 us; error 36 us; skew 500 ppm
Sep 29 18:00:57 dist-test-slave-r7ph krb5kdc[22994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704857, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:57.557279 23007 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 18:00:57.558471 23007 webserver.cc:460] Webserver started at http://127.7.118.62:36921/ using document root <none> and password file <none>
I20260929 18:00:57.559058 23007 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:57.559123 23007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:00:57.559300 23007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:00:57.561519 23007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "1b69fa140f0a46608927fcf893789cca"
format_stamp: "Formatted at 2026-09-29 18:00:57 on dist-test-slave-r7ph"
server_key: "3e0f36233ea7ad931f835d7c8dacd246"
server_key_iv: "85f6c2a709dfcd0d90204dbde13388bd"
server_key_version: "encryptionkey@0"
I20260929 18:00:57.562142 23007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "1b69fa140f0a46608927fcf893789cca"
format_stamp: "Formatted at 2026-09-29 18:00:57 on dist-test-slave-r7ph"
server_key: "3e0f36233ea7ad931f835d7c8dacd246"
server_key_iv: "85f6c2a709dfcd0d90204dbde13388bd"
server_key_version: "encryptionkey@0"
I20260929 18:00:57.566497 23007 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 18:00:57.569660 23022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:57.571233 23007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:00:57.571399 23007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "1b69fa140f0a46608927fcf893789cca"
format_stamp: "Formatted at 2026-09-29 18:00:57 on dist-test-slave-r7ph"
server_key: "3e0f36233ea7ad931f835d7c8dacd246"
server_key_iv: "85f6c2a709dfcd0d90204dbde13388bd"
server_key_version: "encryptionkey@0"
I20260929 18:00:57.571542 23007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:57.597395 23007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:57.601377 23007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:57.601612 23007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:57.609089 23007 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:45259
I20260929 18:00:57.609417 23074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:45259 every 8 connection(s)
I20260929 18:00:57.610162 23007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:00:57.610546 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 23007
I20260929 18:00:57.610682 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:00:57.611034 7640 external_mini_cluster.cc:1428] Setting key 14251c09148d87b935a97756a786f86c
I20260929 18:00:57.613523 23075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:00:57.619483 23075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca: Bootstrap starting.
Sep 29 18:00:57 dist-test-slave-r7ph krb5kdc[22994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:00:57.621871 23075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca: Neither blocks nor log segments found. Creating new log.
I20260929 18:00:57.622699 23075 log.cc:824] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca: Log is configured to *not* fsync() on all Append() calls
I20260929 18:00:57.624679 23075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca: No bootstrap required, opened a new log
I20260929 18:00:57.625836 23078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:57.612578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44512 (local address 127.7.118.62:45259)
0929 18:00:57.613087 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:57.613105 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:57.613145 (+ 40us) server_negotiation.cc:407] Connection header received
0929 18:00:57.613915 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:57.613934 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:57.614366 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:57.614764 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:57.615621 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:57.616656 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:57.617567 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:57.618075 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:57.618258 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:57.620917 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:57.620922 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:57.620938 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:57.620972 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:57.622879 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:57.623362 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:57.623374 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:57.623380 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:57.623499 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:57.623800 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:57.623804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:57.623805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:57.624234 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:57.624454 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:57.624754 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 18:00:57.625070 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":162,"threads_started":1}
I20260929 18:00:57.629029 23075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69fa140f0a46608927fcf893789cca" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45259 } }
I20260929 18:00:57.629319 23075 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:00:57.629395 23075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b69fa140f0a46608927fcf893789cca, State: Initialized, Role: FOLLOWER
I20260929 18:00:57.629886 23075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69fa140f0a46608927fcf893789cca" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45259 } }
I20260929 18:00:57.630064 23075 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:00:57.630131 23075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:00:57.630228 23075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:00:57.631295 23075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69fa140f0a46608927fcf893789cca" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45259 } }
I20260929 18:00:57.631661 23075 leader_election.cc:302] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b69fa140f0a46608927fcf893789cca; no voters:
I20260929 18:00:57.632006 23075 leader_election.cc:288] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:00:57.632234 23080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:00:57.632494 23080 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [term 1 LEADER]: Becoming Leader. State: Replica: 1b69fa140f0a46608927fcf893789cca, State: Running, Role: LEADER
I20260929 18:00:57.632859 23080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69fa140f0a46608927fcf893789cca" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45259 } }
I20260929 18:00:57.633512 23075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:00:57.634814 23081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b69fa140f0a46608927fcf893789cca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69fa140f0a46608927fcf893789cca" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45259 } } }
I20260929 18:00:57.634950 23081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [sys.catalog]: This master's current role is: LEADER
I20260929 18:00:57.635401 23089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:00:57.636152 23082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b69fa140f0a46608927fcf893789cca. Latest consensus state: current_term: 1 leader_uuid: "1b69fa140f0a46608927fcf893789cca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69fa140f0a46608927fcf893789cca" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45259 } } }
I20260929 18:00:57.636258 23082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [sys.catalog]: This master's current role is: LEADER
I20260929 18:00:57.638834 23089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:00:57.644670 23089 catalog_manager.cc:1347] Generated new cluster ID: 217edaa33a804f368501e0b1335b47b2
I20260929 18:00:57.644760 23089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:00:57.650995 23089 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:00:57.652308 23089 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:00:57.658468 23089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca: Generated new TSK 0
I20260929 18:00:57.659267 23089 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.118.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
I20260929 18:00:57.690440 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:0
--local_ip_for_outbound_sockets=127.7.118.1
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:45259
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:42109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:00:57.808593 23101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:57.808897 23101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:57.808943 23101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:00:57.813352 23101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:57.813441 23101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:57.813534 23101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 18:00:57.819515 23101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:42109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:45259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:57.820734 23101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:57.821842 23101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:00:57.832250 23109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:57.832259 23107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:57.832240 23106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:57.832496 23101 server_base.cc:1034] running on GCE node
I20260929 18:00:57.833114 23101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:57.834013 23101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:57.835227 23101 hybrid_clock.cc:648] HybridClock initialized: now 1790704857835181 us; error 58 us; skew 500 ppm
Sep 29 18:00:57 dist-test-slave-r7ph krb5kdc[22994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704857, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:57.838403 23101 init.cc:377] Logged in from keytab as kudu/127.7.118.1@KRBTEST.COM (short username kudu)
I20260929 18:00:57.839434 23101 webserver.cc:460] Webserver started at http://127.7.118.1:38857/ using document root <none> and password file <none>
I20260929 18:00:57.839989 23101 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:57.840052 23101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:00:57.840235 23101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:00:57.842545 23101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/data/instance:
uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4"
format_stamp: "Formatted at 2026-09-29 18:00:57 on dist-test-slave-r7ph"
server_key: "ccc2b037700dec16a4a609473e0c34d5"
server_key_iv: "ff92e57159fbabb0c005879f29a0b813"
server_key_version: "encryptionkey@0"
I20260929 18:00:57.843165 23101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance:
uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4"
format_stamp: "Formatted at 2026-09-29 18:00:57 on dist-test-slave-r7ph"
server_key: "ccc2b037700dec16a4a609473e0c34d5"
server_key_iv: "ff92e57159fbabb0c005879f29a0b813"
server_key_version: "encryptionkey@0"
I20260929 18:00:57.847286 23101 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:00:57.850389 23116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:57.852488 23101 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:00:57.852674 23101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4"
format_stamp: "Formatted at 2026-09-29 18:00:57 on dist-test-slave-r7ph"
server_key: "ccc2b037700dec16a4a609473e0c34d5"
server_key_iv: "ff92e57159fbabb0c005879f29a0b813"
server_key_version: "encryptionkey@0"
I20260929 18:00:57.852857 23101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:57.883877 23101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:57.887882 23101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:57.888146 23101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:57.888836 23101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:00:57.889765 23101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:00:57.889846 23101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:57.889932 23101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:00:57.889981 23101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:57.901011 23101 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:35227
I20260929 18:00:57.901031 23229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:35227 every 8 connection(s)
I20260929 18:00:57.902038 23101 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 18:00:57.907692 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 23101
I20260929 18:00:57.907804 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance
I20260929 18:00:57.908138 7640 external_mini_cluster.cc:1428] Setting key e6e89a1d5a27c63c8e8c236d14261eff
Sep 29 18:00:57 dist-test-slave-r7ph krb5kdc[22994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704857, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.118.2@KRBTEST.COM; defaulting to no policy
I20260929 18:00:57.917889 23078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:57.904082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:42609 (local address 127.7.118.62:45259)
0929 18:00:57.904239 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:57.904243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:57.905204 (+ 961us) server_negotiation.cc:407] Connection header received
0929 18:00:57.906822 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:57.906826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:57.906890 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:57.907011 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:57.908511 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:57.909218 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:57.910620 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:57.910888 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:57.911016 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:57.914043 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:57.914046 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:57.914049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:57.914074 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:57.915642 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:57.916230 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:57.916234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:57.916237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:57.916320 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:57.916693 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:57.916698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:57.916700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:57.916903 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:57.917016 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:57.917568 (+ 552us) server_negotiation.cc:299] Negotiation successful
0929 18:00:57.917719 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:00:57.919124 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:57.904352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.1:42609)
0929 18:00:57.904938 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:57.904988 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:57.906569 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:57.907210 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:57.907218 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:57.907655 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:00:57.908347 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:57.908366 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:57.909375 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:57.909378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:57.910477 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:57.910488 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:57.911066 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:57.911071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:57.911260 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:57.912030 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:00:57.912049 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:00:57.913825 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:00:57.915789 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:57.915797 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:57.915815 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:57.916105 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:57.916423 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:57.916426 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:57.916428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:57.916595 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:57.917015 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:00:57.917024 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:00:57.917313 (+ 289us) client_negotiation.cc:769] Sending connection context
0929 18:00:57.917538 (+ 225us) client_negotiation.cc:240] Negotiation successful
0929 18:00:57.917948 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":131,"threads_started":1}
I20260929 18:00:57.920543 23230 heartbeater.cc:347] Connected to a master server at 127.7.118.62:45259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.2@KRBTEST.COM" created.
I20260929 18:00:57.920828 23230 heartbeater.cc:464] Registering TS with master...
I20260929 18:00:57.921469 23230 heartbeater.cc:511] Master 127.7.118.62:45259 requested a full tablet report, sending...
I20260929 18:00:57.923609 23039 ts_manager.cc:194] Registered new tserver with Master: 78e7cc9c944e49d3af0fd3a1c5f4d7c4 (127.7.118.1:35227)
I20260929 18:00:57.925550 23039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.1@KRBTEST.COM'} at 127.7.118.1:42609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
I20260929 18:00:57.938383 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:0
--local_ip_for_outbound_sockets=127.7.118.2
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:45259
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:42109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:00:58.055729 23235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:58.056026 23235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:58.056073 23235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:00:58.060518 23235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:58.060590 23235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:58.060654 23235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 18:00:58.066180 23235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:42109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:45259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:58.067348 23235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:58.068361 23235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:00:58.078693 23240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:58.078696 23241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:58.078739 23235 server_base.cc:1034] running on GCE node
W20260929 18:00:58.078696 23243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:58.079533 23235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:58.080224 23235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:58.081444 23235 hybrid_clock.cc:648] HybridClock initialized: now 1790704858081401 us; error 67 us; skew 500 ppm
Sep 29 18:00:58 dist-test-slave-r7ph krb5kdc[22994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704858, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:58.084581 23235 init.cc:377] Logged in from keytab as kudu/127.7.118.2@KRBTEST.COM (short username kudu)
I20260929 18:00:58.085543 23235 webserver.cc:460] Webserver started at http://127.7.118.2:39091/ using document root <none> and password file <none>
I20260929 18:00:58.086131 23235 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:58.086198 23235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:00:58.086450 23235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:00:58.088714 23235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/data/instance:
uuid: "8ba1806660e74869bf5d6276ed729e4f"
format_stamp: "Formatted at 2026-09-29 18:00:58 on dist-test-slave-r7ph"
server_key: "cc99cc47630baed6b31a49ef7b9f6b2f"
server_key_iv: "90e7d25ec8d3598be4adaea6736343ff"
server_key_version: "encryptionkey@0"
I20260929 18:00:58.089293 23235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance:
uuid: "8ba1806660e74869bf5d6276ed729e4f"
format_stamp: "Formatted at 2026-09-29 18:00:58 on dist-test-slave-r7ph"
server_key: "cc99cc47630baed6b31a49ef7b9f6b2f"
server_key_iv: "90e7d25ec8d3598be4adaea6736343ff"
server_key_version: "encryptionkey@0"
I20260929 18:00:58.093441 23235 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:00:58.096439 23250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:58.098057 23235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:00:58.098222 23235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "8ba1806660e74869bf5d6276ed729e4f"
format_stamp: "Formatted at 2026-09-29 18:00:58 on dist-test-slave-r7ph"
server_key: "cc99cc47630baed6b31a49ef7b9f6b2f"
server_key_iv: "90e7d25ec8d3598be4adaea6736343ff"
server_key_version: "encryptionkey@0"
I20260929 18:00:58.098402 23235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:58.117944 23235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:58.121752 23235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:58.121984 23235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:58.122731 23235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:00:58.123641 23235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:00:58.123690 23235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:58.123730 23235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:00:58.123786 23235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:58.133451 23235 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:34159
I20260929 18:00:58.133476 23363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:34159 every 8 connection(s)
I20260929 18:00:58.134691 23235 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 18:00:58.135524 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 23235
I20260929 18:00:58.135653 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance
I20260929 18:00:58.135990 7640 external_mini_cluster.cc:1428] Setting key e6b3e66d492184fc993063c551b54105
Sep 29 18:00:58 dist-test-slave-r7ph krb5kdc[22994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704858, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.118.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.3@KRBTEST.COM" created.
I20260929 18:00:58.151767 23078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:58.137078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:48175 (local address 127.7.118.62:45259)
0929 18:00:58.137223 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:58.137228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:58.138510 (+ 1282us) server_negotiation.cc:407] Connection header received
0929 18:00:58.140324 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:58.140330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:58.140397 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:58.140505 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:58.141931 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.142599 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.143839 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.144077 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.144196 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:58.147307 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:58.147310 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:58.147312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:58.147341 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:58.149047 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:58.149672 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:58.149675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:58.149676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:58.149771 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:58.150342 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:58.150350 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:58.150353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:58.150626 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:58.150729 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:58.151492 (+ 763us) server_negotiation.cc:299] Negotiation successful
0929 18:00:58.151634 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:00:58.152796 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:58.137667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.2:48175)
0929 18:00:58.138329 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:58.138400 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:58.140075 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:58.140675 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:58.140682 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:58.141221 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:00:58.141757 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:58.141771 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.142743 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.142746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:58.143703 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:58.143713 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.144250 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.144253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:58.144436 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:58.145132 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:00:58.145149 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:00:58.147085 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:00:58.149219 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:58.149225 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:58.149239 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:58.149550 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:58.149906 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:58.149908 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:58.149910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:58.150165 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:58.150759 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:00:58.150766 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:00:58.151178 (+ 412us) client_negotiation.cc:769] Sending connection context
0929 18:00:58.151404 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 18:00:58.151819 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":152,"threads_started":1}
I20260929 18:00:58.154352 23364 heartbeater.cc:347] Connected to a master server at 127.7.118.62:45259
I20260929 18:00:58.154633 23364 heartbeater.cc:464] Registering TS with master...
I20260929 18:00:58.155258 23364 heartbeater.cc:511] Master 127.7.118.62:45259 requested a full tablet report, sending...
I20260929 18:00:58.156526 23039 ts_manager.cc:194] Registered new tserver with Master: 8ba1806660e74869bf5d6276ed729e4f (127.7.118.2:34159)
I20260929 18:00:58.157196 23039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.2@KRBTEST.COM'} at 127.7.118.2:48175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
I20260929 18:00:58.171025 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:0
--local_ip_for_outbound_sockets=127.7.118.3
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:45259
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:42109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:00:58.299289 23369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:00:58.299610 23369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:00:58.299697 23369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:00:58.304241 23369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:00:58.304322 23369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:00:58.304388 23369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 18:00:58.310169 23369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:42109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:45259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:00:58.311398 23369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:00:58.312507 23369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:00:58.322903 23375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:58.322981 23377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:00:58.322903 23374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:00:58.323109 23369 server_base.cc:1034] running on GCE node
I20260929 18:00:58.323750 23369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:00:58.324483 23369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:00:58.325671 23369 hybrid_clock.cc:648] HybridClock initialized: now 1790704858325629 us; error 60 us; skew 500 ppm
Sep 29 18:00:58 dist-test-slave-r7ph krb5kdc[22994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704858, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:00:58.328774 23369 init.cc:377] Logged in from keytab as kudu/127.7.118.3@KRBTEST.COM (short username kudu)
I20260929 18:00:58.329766 23369 webserver.cc:460] Webserver started at http://127.7.118.3:40131/ using document root <none> and password file <none>
I20260929 18:00:58.330452 23369 fs_manager.cc:359] Metadata directory not provided
I20260929 18:00:58.330523 23369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:00:58.330780 23369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:00:58.332931 23369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/data/instance:
uuid: "5275fffe40da4606b4e58ec633861a94"
format_stamp: "Formatted at 2026-09-29 18:00:58 on dist-test-slave-r7ph"
server_key: "469e4549a2d26be67494466092c123bf"
server_key_iv: "99a7ccee176155639d6c1c7b4cd3ac76"
server_key_version: "encryptionkey@0"
I20260929 18:00:58.333588 23369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance:
uuid: "5275fffe40da4606b4e58ec633861a94"
format_stamp: "Formatted at 2026-09-29 18:00:58 on dist-test-slave-r7ph"
server_key: "469e4549a2d26be67494466092c123bf"
server_key_iv: "99a7ccee176155639d6c1c7b4cd3ac76"
server_key_version: "encryptionkey@0"
I20260929 18:00:58.337710 23369 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 18:00:58.340447 23384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:58.341905 23369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:00:58.342037 23369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "5275fffe40da4606b4e58ec633861a94"
format_stamp: "Formatted at 2026-09-29 18:00:58 on dist-test-slave-r7ph"
server_key: "469e4549a2d26be67494466092c123bf"
server_key_iv: "99a7ccee176155639d6c1c7b4cd3ac76"
server_key_version: "encryptionkey@0"
I20260929 18:00:58.342199 23369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:00:58.355572 23369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:00:58.359012 23369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:00:58.359259 23369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:00:58.359930 23369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:00:58.360869 23369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:00:58.360914 23369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:58.360954 23369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:00:58.361004 23369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:00:58.370972 23369 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:45609
I20260929 18:00:58.370992 23497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:45609 every 8 connection(s)
I20260929 18:00:58.372016 23369 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 18:00:58.378923 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 23369
I20260929 18:00:58.379034 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance
I20260929 18:00:58.379350 7640 external_mini_cluster.cc:1428] Setting key 6cb46f6388f841cc5ebe6c4ab8eb0995
Sep 29 18:00:58 dist-test-slave-r7ph krb5kdc[22994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704858, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:00:58.387463 23078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:58.374166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:42455 (local address 127.7.118.62:45259)
0929 18:00:58.374354 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:58.374360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:58.375520 (+ 1160us) server_negotiation.cc:407] Connection header received
0929 18:00:58.377040 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:58.377043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:58.377106 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:58.377218 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:58.378517 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.379166 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.380435 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.380664 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.380776 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:58.383638 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:58.383642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:58.383644 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:58.383672 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:58.385205 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:58.385806 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:58.385810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:58.385811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:58.385897 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:58.386290 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:58.386293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:58.386295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:58.386496 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:58.386604 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:58.387160 (+ 556us) server_negotiation.cc:299] Negotiation successful
0929 18:00:58.387295 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 18:00:58.388441 23500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:58.374742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.3:42455)
0929 18:00:58.375252 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:58.375343 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:58.376798 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:58.377417 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:58.377425 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:58.377820 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:00:58.378352 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:58.378375 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.379300 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.379304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:58.380284 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:58.380294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.380833 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.380837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:58.381079 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:58.381750 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:00:58.381768 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:00:58.383434 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:00:58.385349 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:58.385369 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:58.385390 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:58.385681 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:58.385995 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:00:58.385999 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:00:58.386001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:00:58.386162 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:00:58.386615 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:00:58.386624 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:00:58.386912 (+ 288us) client_negotiation.cc:769] Sending connection context
0929 18:00:58.387105 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 18:00:58.387479 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":134,"threads_started":1}
I20260929 18:00:58.389671 23498 heartbeater.cc:347] Connected to a master server at 127.7.118.62:45259
I20260929 18:00:58.389950 23498 heartbeater.cc:464] Registering TS with master...
I20260929 18:00:58.390530 23498 heartbeater.cc:511] Master 127.7.118.62:45259 requested a full tablet report, sending...
I20260929 18:00:58.391602 23039 ts_manager.cc:194] Registered new tserver with Master: 5275fffe40da4606b4e58ec633861a94 (127.7.118.3:45609)
I20260929 18:00:58.392331 23039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.3@KRBTEST.COM'} at 127.7.118.3:42455
I20260929 18:00:58.393497 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:00:58.403160 23078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:58.395520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44528 (local address 127.7.118.62:45259)
0929 18:00:58.395658 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:58.395662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:58.395823 (+ 161us) server_negotiation.cc:407] Connection header received
0929 18:00:58.395932 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:58.395934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:58.395971 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:58.396038 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:00:58.396759 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.397269 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.398133 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.398410 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.398495 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:58.399975 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:00:58.399978 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:00:58.399980 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:00:58.399999 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:00:58.401344 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:58.401773 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:58.401775 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:58.401777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:58.401846 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:00:58.402146 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:00:58.402148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:00:58.402149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:00:58.402463 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:00:58.402622 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:58.402854 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 18:00:58.402971 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:00:58.406503 23039 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:00:58.409551 23039 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:00:58.424600 23509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:58.418406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:45609 (local address 127.0.0.1:59564)
0929 18:00:58.418817 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:58.419553 (+ 736us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:58.419660 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:58.420455 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:58.420461 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:58.420484 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:00:58.420601 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:58.420608 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.422019 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.422022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:58.423135 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:58.423145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.424013 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.424016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:58.424134 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:58.424171 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 18:00:58.424311 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 18:00:58.424446 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":18,"thread_start_us":91,"threads_started":1}
I20260929 18:00:58.424610 23510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:58.418563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59564 (local address 127.7.118.3:45609)
0929 18:00:58.419896 (+ 1333us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:58.419912 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:58.419936 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:00:58.420015 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:58.420020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:58.420266 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:58.420450 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:00:58.420715 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.421866 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.423257 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.423849 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.424001 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:58.424167 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:58.424341 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 18:00:58.424445 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":137,"threads_started":1}
I20260929 18:00:58.425606 23511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:58.418621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:34159 (local address 127.0.0.1:34132)
0929 18:00:58.419290 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:58.419340 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:58.419464 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:58.420206 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:58.420212 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:58.420235 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:00:58.420390 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:58.420420 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.421632 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.421637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:58.423912 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:58.423922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.425222 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.425227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:58.425321 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:58.425355 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 18:00:58.425422 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 18:00:58.425494 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":122,"thread_start_us":116,"threads_started":1}
I20260929 18:00:58.425951 23514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:58.419698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39458 (local address 127.7.118.1:35227)
0929 18:00:58.420106 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:58.420112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:58.420433 (+ 321us) server_negotiation.cc:407] Connection header received
0929 18:00:58.420510 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:58.420515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:58.420725 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:58.420866 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:00:58.421082 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.422395 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.424072 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.424897 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.425007 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:58.425152 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:58.425699 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 18:00:58.425809 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":74,"threads_started":1}
I20260929 18:00:58.426064 23513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:58.419305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34132 (local address 127.7.118.2:34159)
0929 18:00:58.419700 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:58.419713 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:58.419738 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:00:58.419824 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:58.419829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:58.420037 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:58.420206 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:00:58.420549 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.421501 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.424061 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.424897 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.425007 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:58.425152 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:58.425801 (+ 649us) server_negotiation.cc:299] Negotiation successful
0929 18:00:58.425927 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":87,"threads_started":1}
I20260929 18:00:58.426185 23512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:58.419308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:35227 (local address 127.0.0.1:39458)
0929 18:00:58.420230 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:58.420243 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:58.420334 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:58.420865 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:58.420868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:58.420886 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:00:58.420964 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:58.420970 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.422490 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.422494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:58.423912 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:58.423922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.425022 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.425026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:58.425119 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:58.425148 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 18:00:58.425223 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 18:00:58.425295 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"mutex_wait_us":116,"thread_start_us":95,"threads_started":1}
I20260929 18:00:58.428256 23432 tablet_service.cc:1467] Processing CreateTablet for tablet 915571735c7e40afbd5c9d2c1613e9bf (DEFAULT_TABLE table=test-table [id=a7a9420d5ec943a79a6bae93a8f68ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:00:58.428826 23164 tablet_service.cc:1467] Processing CreateTablet for tablet 915571735c7e40afbd5c9d2c1613e9bf (DEFAULT_TABLE table=test-table [id=a7a9420d5ec943a79a6bae93a8f68ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:00:58.428826 23298 tablet_service.cc:1467] Processing CreateTablet for tablet 915571735c7e40afbd5c9d2c1613e9bf (DEFAULT_TABLE table=test-table [id=a7a9420d5ec943a79a6bae93a8f68ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:00:58.429318 23432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915571735c7e40afbd5c9d2c1613e9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:00:58.429862 23298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915571735c7e40afbd5c9d2c1613e9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:00:58.429862 23164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915571735c7e40afbd5c9d2c1613e9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:00:58.436254 23515 tablet_bootstrap.cc:492] T 915571735c7e40afbd5c9d2c1613e9bf P 5275fffe40da4606b4e58ec633861a94: Bootstrap starting.
I20260929 18:00:58.436975 23516 tablet_bootstrap.cc:492] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Bootstrap starting.
I20260929 18:00:58.438568 23517 tablet_bootstrap.cc:492] T 915571735c7e40afbd5c9d2c1613e9bf P 8ba1806660e74869bf5d6276ed729e4f: Bootstrap starting.
I20260929 18:00:58.439495 23516 tablet_bootstrap.cc:654] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Neither blocks nor log segments found. Creating new log.
I20260929 18:00:58.439925 23515 tablet_bootstrap.cc:654] T 915571735c7e40afbd5c9d2c1613e9bf P 5275fffe40da4606b4e58ec633861a94: Neither blocks nor log segments found. Creating new log.
I20260929 18:00:58.440522 23516 log.cc:824] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Log is configured to *not* fsync() on all Append() calls
I20260929 18:00:58.440747 23515 log.cc:824] T 915571735c7e40afbd5c9d2c1613e9bf P 5275fffe40da4606b4e58ec633861a94: Log is configured to *not* fsync() on all Append() calls
I20260929 18:00:58.440927 23517 tablet_bootstrap.cc:654] T 915571735c7e40afbd5c9d2c1613e9bf P 8ba1806660e74869bf5d6276ed729e4f: Neither blocks nor log segments found. Creating new log.
I20260929 18:00:58.441744 23517 log.cc:824] T 915571735c7e40afbd5c9d2c1613e9bf P 8ba1806660e74869bf5d6276ed729e4f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:00:58.442903 23516 tablet_bootstrap.cc:492] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: No bootstrap required, opened a new log
I20260929 18:00:58.442963 23515 tablet_bootstrap.cc:492] T 915571735c7e40afbd5c9d2c1613e9bf P 5275fffe40da4606b4e58ec633861a94: No bootstrap required, opened a new log
I20260929 18:00:58.443140 23516 ts_tablet_manager.cc:1397] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:00:58.443146 23515 ts_tablet_manager.cc:1397] T 915571735c7e40afbd5c9d2c1613e9bf P 5275fffe40da4606b4e58ec633861a94: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 18:00:58.443753 23517 tablet_bootstrap.cc:492] T 915571735c7e40afbd5c9d2c1613e9bf P 8ba1806660e74869bf5d6276ed729e4f: No bootstrap required, opened a new log
I20260929 18:00:58.443926 23517 ts_tablet_manager.cc:1397] T 915571735c7e40afbd5c9d2c1613e9bf P 8ba1806660e74869bf5d6276ed729e4f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:00:58.446905 23516 raft_consensus.cc:348] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:00:58.447258 23516 raft_consensus.cc:374] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:00:58.447319 23516 raft_consensus.cc:729] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e7cc9c944e49d3af0fd3a1c5f4d7c4, State: Initialized, Role: FOLLOWER
I20260929 18:00:58.447189 23515 raft_consensus.cc:348] T 915571735c7e40afbd5c9d2c1613e9bf P 5275fffe40da4606b4e58ec633861a94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:00:58.447474 23515 raft_consensus.cc:374] T 915571735c7e40afbd5c9d2c1613e9bf P 5275fffe40da4606b4e58ec633861a94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:00:58.447566 23515 raft_consensus.cc:729] T 915571735c7e40afbd5c9d2c1613e9bf P 5275fffe40da4606b4e58ec633861a94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5275fffe40da4606b4e58ec633861a94, State: Initialized, Role: FOLLOWER
I20260929 18:00:58.447790 23516 consensus_queue.cc:260] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:00:58.447988 23515 consensus_queue.cc:260] T 915571735c7e40afbd5c9d2c1613e9bf P 5275fffe40da4606b4e58ec633861a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:00:58.447937 23517 raft_consensus.cc:348] T 915571735c7e40afbd5c9d2c1613e9bf P 8ba1806660e74869bf5d6276ed729e4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:00:58.448201 23517 raft_consensus.cc:374] T 915571735c7e40afbd5c9d2c1613e9bf P 8ba1806660e74869bf5d6276ed729e4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:00:58.448261 23517 raft_consensus.cc:729] T 915571735c7e40afbd5c9d2c1613e9bf P 8ba1806660e74869bf5d6276ed729e4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ba1806660e74869bf5d6276ed729e4f, State: Initialized, Role: FOLLOWER
I20260929 18:00:58.448689 23517 consensus_queue.cc:260] T 915571735c7e40afbd5c9d2c1613e9bf P 8ba1806660e74869bf5d6276ed729e4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:00:58.448948 23516 ts_tablet_manager.cc:1428] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:00:58.448989 23515 ts_tablet_manager.cc:1428] T 915571735c7e40afbd5c9d2c1613e9bf P 5275fffe40da4606b4e58ec633861a94: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:00:58.449337 23498 heartbeater.cc:503] Master 127.7.118.62:45259 was elected leader, sending a full tablet report...
I20260929 18:00:58.449337 23230 heartbeater.cc:503] Master 127.7.118.62:45259 was elected leader, sending a full tablet report...
I20260929 18:00:58.449501 23364 heartbeater.cc:503] Master 127.7.118.62:45259 was elected leader, sending a full tablet report...
I20260929 18:00:58.450977 23517 ts_tablet_manager.cc:1428] T 915571735c7e40afbd5c9d2c1613e9bf P 8ba1806660e74869bf5d6276ed729e4f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260929 18:00:58.623178 23499 tablet.cc:2369] T 915571735c7e40afbd5c9d2c1613e9bf P 5275fffe40da4606b4e58ec633861a94: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:00:58.636327 23365 tablet.cc:2369] T 915571735c7e40afbd5c9d2c1613e9bf P 8ba1806660e74869bf5d6276ed729e4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:00:58.643838 23521 raft_consensus.cc:482] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:00:58.644109 23521 raft_consensus.cc:504] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:00:58.645629 23521 leader_election.cc:288] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5275fffe40da4606b4e58ec633861a94 (127.7.118.3:45609), 8ba1806660e74869bf5d6276ed729e4f (127.7.118.2:34159)
I20260929 18:00:58.650049 23510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:58.646020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:51679 (local address 127.7.118.3:45609)
0929 18:00:58.646179 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:58.646184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:58.646532 (+ 348us) server_negotiation.cc:407] Connection header received
0929 18:00:58.646723 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:58.646727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:58.646778 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:58.646859 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:00:58.647197 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.647711 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.648988 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.649409 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.649535 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:58.649610 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:58.649827 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 18:00:58.649911 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:00:58.650104 23525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:58.645885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:34159 (local address 127.7.118.1:55121)
0929 18:00:58.646385 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:58.646403 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:58.646591 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:58.646881 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:58.646884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:58.646927 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:00:58.647082 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:58.647088 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.647863 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.647867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:58.648855 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:58.648865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.649547 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.649550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:58.649713 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:58.649731 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 18:00:58.649846 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 18:00:58.649934 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":66,"spinlock_wait_cycles":30208,"thread_start_us":135,"threads_started":1}
I20260929 18:00:58.650112 23524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:58.645896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:45609 (local address 127.7.118.1:51679)
0929 18:00:58.646433 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:58.646450 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:00:58.646600 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:00:58.646882 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:00:58.646885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:00:58.646930 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:00:58.647084 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:58.647090 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.647830 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.647836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:58.648845 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:00:58.648857 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.649547 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.649551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:00:58.649675 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:58.649709 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 18:00:58.649811 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 18:00:58.649892 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":130,"spinlock_wait_cycles":4864,"thread_start_us":119,"threads_started":1}
I20260929 18:00:58.650071 23513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:58.645999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:55121 (local address 127.7.118.2:34159)
0929 18:00:58.646230 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:58.646235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:58.646504 (+ 269us) server_negotiation.cc:407] Connection header received
0929 18:00:58.646710 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:58.646716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:58.646778 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:58.646870 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:00:58.647235 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.647740 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.648988 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.649412 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.649538 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:58.649614 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:58.649868 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 18:00:58.649943 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260929 18:00:58.650988 23452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915571735c7e40afbd5c9d2c1613e9bf" candidate_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5275fffe40da4606b4e58ec633861a94" is_pre_election: true
I20260929 18:00:58.651223 23318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915571735c7e40afbd5c9d2c1613e9bf" candidate_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ba1806660e74869bf5d6276ed729e4f" is_pre_election: true
I20260929 18:00:58.651355 23452 raft_consensus.cc:2454] T 915571735c7e40afbd5c9d2c1613e9bf P 5275fffe40da4606b4e58ec633861a94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e7cc9c944e49d3af0fd3a1c5f4d7c4 in term 0.
I20260929 18:00:58.651569 23318 raft_consensus.cc:2454] T 915571735c7e40afbd5c9d2c1613e9bf P 8ba1806660e74869bf5d6276ed729e4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e7cc9c944e49d3af0fd3a1c5f4d7c4 in term 0.
I20260929 18:00:58.651952 23119 leader_election.cc:302] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5275fffe40da4606b4e58ec633861a94, 78e7cc9c944e49d3af0fd3a1c5f4d7c4; no voters:
I20260929 18:00:58.652275 23521 raft_consensus.cc:2781] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:00:58.652371 23521 raft_consensus.cc:482] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:00:58.652416 23521 raft_consensus.cc:3037] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 0 FOLLOWER]: Advancing to term 1
W20260929 18:00:58.653604 23231 tablet.cc:2369] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:00:58.653734 23521 raft_consensus.cc:504] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:00:58.654544 23521 leader_election.cc:288] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 1 election: Requested vote from peers 5275fffe40da4606b4e58ec633861a94 (127.7.118.3:45609), 8ba1806660e74869bf5d6276ed729e4f (127.7.118.2:34159)
I20260929 18:00:58.654644 23452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915571735c7e40afbd5c9d2c1613e9bf" candidate_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5275fffe40da4606b4e58ec633861a94"
I20260929 18:00:58.654619 23318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915571735c7e40afbd5c9d2c1613e9bf" candidate_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ba1806660e74869bf5d6276ed729e4f"
I20260929 18:00:58.654788 23452 raft_consensus.cc:3037] T 915571735c7e40afbd5c9d2c1613e9bf P 5275fffe40da4606b4e58ec633861a94 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:00:58.654798 23318 raft_consensus.cc:3037] T 915571735c7e40afbd5c9d2c1613e9bf P 8ba1806660e74869bf5d6276ed729e4f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:00:58.655864 23452 raft_consensus.cc:2454] T 915571735c7e40afbd5c9d2c1613e9bf P 5275fffe40da4606b4e58ec633861a94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e7cc9c944e49d3af0fd3a1c5f4d7c4 in term 1.
I20260929 18:00:58.655903 23318 raft_consensus.cc:2454] T 915571735c7e40afbd5c9d2c1613e9bf P 8ba1806660e74869bf5d6276ed729e4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e7cc9c944e49d3af0fd3a1c5f4d7c4 in term 1.
I20260929 18:00:58.656263 23119 leader_election.cc:302] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5275fffe40da4606b4e58ec633861a94, 78e7cc9c944e49d3af0fd3a1c5f4d7c4, 8ba1806660e74869bf5d6276ed729e4f; no voters:
I20260929 18:00:58.656463 23521 raft_consensus.cc:2781] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:00:58.656724 23521 raft_consensus.cc:686] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 LEADER]: Becoming Leader. State: Replica: 78e7cc9c944e49d3af0fd3a1c5f4d7c4, State: Running, Role: LEADER
I20260929 18:00:58.657091 23521 consensus_queue.cc:237] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:00:58.661358 23039 catalog_manager.cc:5495] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78e7cc9c944e49d3af0fd3a1c5f4d7c4 (127.7.118.1). New cstate: current_term: 1 leader_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } health_report { overall_health: HEALTHY } } }
I20260929 18:00:58.721329 23514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:58.717352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39474 (local address 127.7.118.1:35227)
0929 18:00:58.717508 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 18:00:58.717521 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:00:58.717534 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:00:58.717619 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:00:58.717621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:00:58.717679 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:00:58.717770 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:00:58.718040 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.718570 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.719501 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:00:58.719726 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:00:58.719819 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:00:58.720180 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:00:58.720658 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:00:58.720768 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 18:00:58.721035 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 18:00:58.721151 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 18:00:58.728403 23452 raft_consensus.cc:1261] T 915571735c7e40afbd5c9d2c1613e9bf P 5275fffe40da4606b4e58ec633861a94 [term 1 FOLLOWER]: Refusing update from remote peer 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:00:58.728422 23318 raft_consensus.cc:1261] T 915571735c7e40afbd5c9d2c1613e9bf P 8ba1806660e74869bf5d6276ed729e4f [term 1 FOLLOWER]: Refusing update from remote peer 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:00:58.729331 23526 consensus_queue.cc:1035] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:00:58.729547 23521 consensus_queue.cc:1035] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:00:58.739792 23531 mvcc.cc:205] Tried to move back new op lower bound from 7334727101342339072 to 7334727101062029312. Current Snapshot: MvccSnapshot[applied={T|T < 7334727101342339072}]
I20260929 18:00:58.750401 23037 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:00:58.750638 23037 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:00:58.753624 23037 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca: Sending DeleteTablet for 3 replicas of tablet 915571735c7e40afbd5c9d2c1613e9bf
I20260929 18:00:58.754724 23298 tablet_service.cc:1514] Processing DeleteTablet for tablet 915571735c7e40afbd5c9d2c1613e9bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:00:58 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:34132
I20260929 18:00:58.754750 23432 tablet_service.cc:1514] Processing DeleteTablet for tablet 915571735c7e40afbd5c9d2c1613e9bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:00:58 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:59564
I20260929 18:00:58.754916 23164 tablet_service.cc:1514] Processing DeleteTablet for tablet 915571735c7e40afbd5c9d2c1613e9bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:00:58 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:39458
I20260929 18:00:58.755324 23540 tablet_replica.cc:330] T 915571735c7e40afbd5c9d2c1613e9bf P 5275fffe40da4606b4e58ec633861a94: stopping tablet replica
I20260929 18:00:58.755573 23541 tablet_replica.cc:330] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: stopping tablet replica
I20260929 18:00:58.755771 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 23101
I20260929 18:00:58.755934 23540 raft_consensus.cc:2229] T 915571735c7e40afbd5c9d2c1613e9bf P 5275fffe40da4606b4e58ec633861a94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:00:58.756037 23541 raft_consensus.cc:2229] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:00:58.756305 23540 raft_consensus.cc:2258] T 915571735c7e40afbd5c9d2c1613e9bf P 5275fffe40da4606b4e58ec633861a94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:00:58.756635 23539 tablet_replica.cc:330] T 915571735c7e40afbd5c9d2c1613e9bf P 8ba1806660e74869bf5d6276ed729e4f: stopping tablet replica
I20260929 18:00:58.757010 23539 raft_consensus.cc:2229] T 915571735c7e40afbd5c9d2c1613e9bf P 8ba1806660e74869bf5d6276ed729e4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:00:58.757377 23539 raft_consensus.cc:2258] T 915571735c7e40afbd5c9d2c1613e9bf P 8ba1806660e74869bf5d6276ed729e4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:00:58.757923 23540 ts_tablet_manager.cc:1905] T 915571735c7e40afbd5c9d2c1613e9bf P 5275fffe40da4606b4e58ec633861a94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:00:58.760063 23539 ts_tablet_manager.cc:1905] T 915571735c7e40afbd5c9d2c1613e9bf P 8ba1806660e74869bf5d6276ed729e4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:00:58.762449 23540 ts_tablet_manager.cc:1918] T 915571735c7e40afbd5c9d2c1613e9bf P 5275fffe40da4606b4e58ec633861a94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:00:58.762650 23539 ts_tablet_manager.cc:1918] T 915571735c7e40afbd5c9d2c1613e9bf P 8ba1806660e74869bf5d6276ed729e4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:00:58.762712 23540 log.cc:1196] T 915571735c7e40afbd5c9d2c1613e9bf P 5275fffe40da4606b4e58ec633861a94: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/wal/wals/915571735c7e40afbd5c9d2c1613e9bf
I20260929 18:00:58.762797 23539 log.cc:1196] T 915571735c7e40afbd5c9d2c1613e9bf P 8ba1806660e74869bf5d6276ed729e4f: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/wal/wals/915571735c7e40afbd5c9d2c1613e9bf
I20260929 18:00:58.763123 23539 ts_tablet_manager.cc:1939] T 915571735c7e40afbd5c9d2c1613e9bf P 8ba1806660e74869bf5d6276ed729e4f: Deleting consensus metadata
I20260929 18:00:58.763326 23540 ts_tablet_manager.cc:1939] T 915571735c7e40afbd5c9d2c1613e9bf P 5275fffe40da4606b4e58ec633861a94: Deleting consensus metadata
I20260929 18:00:58.764598 23024 catalog_manager.cc:4841] TS 8ba1806660e74869bf5d6276ed729e4f (127.7.118.2:34159): tablet 915571735c7e40afbd5c9d2c1613e9bf (table test-table [id=a7a9420d5ec943a79a6bae93a8f68ec7]) successfully deleted
I20260929 18:00:58.765014 23025 catalog_manager.cc:4841] TS 5275fffe40da4606b4e58ec633861a94 (127.7.118.3:45609): tablet 915571735c7e40afbd5c9d2c1613e9bf (table test-table [id=a7a9420d5ec943a79a6bae93a8f68ec7]) successfully deleted
W20260929 18:00:58.765396 23025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.118.1:35227 (error 108)
I20260929 18:00:58.765733 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 23235
I20260929 18:00:58.766983 23512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:00:58.766647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:35227 (local address 127.0.0.1:39486)
0929 18:00:58.766823 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 18:00:58.766904 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.1:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 18:00:58.767285 23025 catalog_manager.cc:4584] TS 78e7cc9c944e49d3af0fd3a1c5f4d7c4 (127.7.118.1:35227): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 915571735c7e40afbd5c9d2c1613e9bf: Network error: Client connection negotiation failed: client connection to 127.7.118.1:35227: connect: Connection refused (error 111)
I20260929 18:00:58.772989 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 23369
I20260929 18:00:58.779454 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 23007
I20260929 18:01:18.787175 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:45259
--webserver_interface=127.7.118.62
--webserver_port=36921
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:42109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:45259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:01:18.905835 23543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:01:18.906306 23543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:01:18.906364 23543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:01:18.910718 23543 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 18:01:18.910796 23543 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 18:01:18.910812 23543 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 18:01:18.910827 23543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:01:18.910846 23543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:01:18.910864 23543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:01:18.910879 23543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:01:18.910895 23543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:01:18.916615 23543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:42109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:45259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:45259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=36921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:01:18.917773 23543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:01:18.918910 23543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:01:18.933933 23543 server_base.cc:1034] running on GCE node
W20260929 18:01:18.933868 23552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:01:18.933868 23549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:01:18.933868 23550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:01:18.934895 23543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:01:18.936200 23543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:01:18.937422 23543 hybrid_clock.cc:648] HybridClock initialized: now 1790704878937407 us; error 38 us; skew 500 ppm
Sep 29 18:01:18 dist-test-slave-r7ph krb5kdc[22994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704878, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:01:18.940727 23543 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 18:01:18.941812 23543 webserver.cc:460] Webserver started at http://127.7.118.62:36921/ using document root <none> and password file <none>
I20260929 18:01:18.942534 23543 fs_manager.cc:359] Metadata directory not provided
I20260929 18:01:18.942653 23543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:01:18.947134 23543 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:01:18.949575 23559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:18.951033 23543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:01:18.951195 23543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "1b69fa140f0a46608927fcf893789cca"
format_stamp: "Formatted at 2026-09-29 18:00:57 on dist-test-slave-r7ph"
server_key: "3e0f36233ea7ad931f835d7c8dacd246"
server_key_iv: "85f6c2a709dfcd0d90204dbde13388bd"
server_key_version: "encryptionkey@0"
I20260929 18:01:18.951975 23543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:01:18.971405 23543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:01:18.975181 23543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:01:18.975412 23543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:01:18.983275 23543 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:45259
I20260929 18:01:18.983595 23611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:45259 every 8 connection(s)
I20260929 18:01:18.984608 23543 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:01:18.989565 23612 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:01:18.991556 23612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca: Bootstrap starting.
I20260929 18:01:18.994571 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 23543
I20260929 18:01:18.995121 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:35227
--local_ip_for_outbound_sockets=127.7.118.1
--tserver_master_addrs=127.7.118.62:45259
--webserver_port=38857
--webserver_interface=127.7.118.1
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:42109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:01:19.011780 23612 log.cc:824] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca: Log is configured to *not* fsync() on all Append() calls
I20260929 18:01:19.021446 23612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:01:19.021893 23612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca: Bootstrap complete.
I20260929 18:01:19.026702 23612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69fa140f0a46608927fcf893789cca" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45259 } }
I20260929 18:01:19.027226 23612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b69fa140f0a46608927fcf893789cca, State: Initialized, Role: FOLLOWER
I20260929 18:01:19.027796 23612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69fa140f0a46608927fcf893789cca" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45259 } }
I20260929 18:01:19.027969 23612 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:01:19.028066 23612 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:01:19.028182 23612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:01:19.030473 23612 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69fa140f0a46608927fcf893789cca" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45259 } }
I20260929 18:01:19.030844 23612 leader_election.cc:302] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b69fa140f0a46608927fcf893789cca; no voters:
I20260929 18:01:19.031198 23612 leader_election.cc:288] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:01:19.031296 23616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:01:19.031502 23616 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [term 2 LEADER]: Becoming Leader. State: Replica: 1b69fa140f0a46608927fcf893789cca, State: Running, Role: LEADER
I20260929 18:01:19.031819 23616 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69fa140f0a46608927fcf893789cca" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45259 } }
I20260929 18:01:19.032595 23612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:01:19.033430 23617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1b69fa140f0a46608927fcf893789cca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69fa140f0a46608927fcf893789cca" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45259 } } }
I20260929 18:01:19.033452 23618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b69fa140f0a46608927fcf893789cca. Latest consensus state: current_term: 2 leader_uuid: "1b69fa140f0a46608927fcf893789cca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69fa140f0a46608927fcf893789cca" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45259 } } }
I20260929 18:01:19.033596 23618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [sys.catalog]: This master's current role is: LEADER
I20260929 18:01:19.033595 23617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [sys.catalog]: This master's current role is: LEADER
I20260929 18:01:19.037948 23631 catalog_manager.cc:1259] Loaded cluster ID: 217edaa33a804f368501e0b1335b47b2
I20260929 18:01:19.038056 23631 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca: loading cluster ID for follower catalog manager: success
I20260929 18:01:19.040119 23631 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca: acquiring CA information for follower catalog manager: success
I20260929 18:01:19.041514 23631 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:01:19.041685 23624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:01:19.044148 23624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:01:19.044674 23624 catalog_manager.cc:1259] Loaded cluster ID: 217edaa33a804f368501e0b1335b47b2
I20260929 18:01:19.044750 23624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:01:19.045344 23624 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:01:19.045794 23624 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca: Loaded TSK: 0
I20260929 18:01:19.065615 23624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca: Generated new TSK 1
I20260929 18:01:19.065939 23624 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:01:19.139060 23614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:01:19.139374 23614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:01:19.139422 23614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:01:19.143757 23614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:01:19.143837 23614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:01:19.143934 23614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 18:01:19.150748 23614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:42109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:35227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=38857
--tserver_master_addrs=127.7.118.62:45259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:01:19.151943 23614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:01:19.153081 23614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:01:19.165129 23640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:01:19.165314 23614 server_base.cc:1034] running on GCE node
W20260929 18:01:19.165123 23639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:01:19.165529 23642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:01:19.166139 23614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:01:19.166935 23614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:01:19.168140 23614 hybrid_clock.cc:648] HybridClock initialized: now 1790704879168100 us; error 60 us; skew 500 ppm
Sep 29 18:01:19 dist-test-slave-r7ph krb5kdc[22994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704879, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:01:19.171422 23614 init.cc:377] Logged in from keytab as kudu/127.7.118.1@KRBTEST.COM (short username kudu)
I20260929 18:01:19.173043 23614 webserver.cc:460] Webserver started at http://127.7.118.1:38857/ using document root <none> and password file <none>
I20260929 18:01:19.173775 23614 fs_manager.cc:359] Metadata directory not provided
I20260929 18:01:19.173900 23614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:01:19.178501 23614 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:01:19.181198 23649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:19.183382 23614 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:01:19.183583 23614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4"
format_stamp: "Formatted at 2026-09-29 18:00:57 on dist-test-slave-r7ph"
server_key: "ccc2b037700dec16a4a609473e0c34d5"
server_key_iv: "ff92e57159fbabb0c005879f29a0b813"
server_key_version: "encryptionkey@0"
I20260929 18:01:19.184523 23614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:01:19.198699 23614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:01:19.202947 23614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:01:19.203501 23614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:01:19.204306 23614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:01:19.205632 23656 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:01:19.210713 23614 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:01:19.210819 23614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260929 18:01:19.210886 23614 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:01:19.212935 23614 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:01:19.213027 23614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 18:01:19.213320 23656 tablet_bootstrap.cc:492] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Bootstrap starting.
I20260929 18:01:19.224895 23614 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:35227
I20260929 18:01:19.225011 23763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:35227 every 8 connection(s)
I20260929 18:01:19.226509 23614 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 18:01:19.233816 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 23614
I20260929 18:01:19.234457 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:34159
--local_ip_for_outbound_sockets=127.7.118.2
--tserver_master_addrs=127.7.118.62:45259
--webserver_port=39091
--webserver_interface=127.7.118.2
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:42109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:01:19.235884 23656 log.cc:824] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:01:19 dist-test-slave-r7ph krb5kdc[22994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704879, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:01:19.246296 23656 tablet_bootstrap.cc:492] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:01:19.246856 23656 tablet_bootstrap.cc:492] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Bootstrap complete.
I20260929 18:01:19.247354 23766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:19.228956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:37959 (local address 127.7.118.62:45259)
0929 18:01:19.229472 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:19.229497 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:19.229834 (+ 337us) server_negotiation.cc:407] Connection header received
0929 18:01:19.231246 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:19.231265 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:19.231758 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:19.232327 (+ 569us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:01:19.233905 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.235829 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.237727 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.238326 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.238532 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:19.241273 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:01:19.241279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:01:19.241293 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:01:19.241327 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:01:19.243499 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:01:19.244153 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:01:19.244160 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:01:19.244170 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:01:19.244322 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:01:19.244784 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:01:19.244790 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:01:19.244794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:01:19.245197 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:01:19.245343 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:19.245970 (+ 627us) server_negotiation.cc:299] Negotiation successful
0929 18:01:19.246313 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":174,"threads_started":1}
I20260929 18:01:19.247355 23767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:19.228969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.1:37959)
0929 18:01:19.229627 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:19.229689 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:19.230712 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:19.232403 (+ 1691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:01:19.232411 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:01:19.232876 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:01:19.233636 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:19.233663 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.236126 (+ 2463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.236134 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:19.237528 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:19.237542 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.238493 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.238497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:19.238740 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:19.239396 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:01:19.239414 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:01:19.241021 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:01:19.243670 (+ 2649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:01:19.243678 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:01:19.243694 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:01:19.243993 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:01:19.244448 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:01:19.244452 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:01:19.244454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:01:19.244671 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:01:19.245333 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:01:19.245340 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:01:19.245629 (+ 289us) client_negotiation.cc:769] Sending connection context
0929 18:01:19.245832 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 18:01:19.246235 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":136,"threads_started":1}
I20260929 18:01:19.248420 23656 ts_tablet_manager.cc:1397] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Time spent bootstrapping tablet: real 0.035s user 0.022s sys 0.011s
I20260929 18:01:19.250380 23764 heartbeater.cc:347] Connected to a master server at 127.7.118.62:45259
I20260929 18:01:19.250656 23764 heartbeater.cc:464] Registering TS with master...
I20260929 18:01:19.251233 23764 heartbeater.cc:511] Master 127.7.118.62:45259 requested a full tablet report, sending...
I20260929 18:01:19.251669 23656 raft_consensus.cc:348] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:01:19.252276 23656 raft_consensus.cc:729] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e7cc9c944e49d3af0fd3a1c5f4d7c4, State: Initialized, Role: FOLLOWER
I20260929 18:01:19.252866 23656 consensus_queue.cc:260] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:01:19.253984 23656 ts_tablet_manager.cc:1428] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:01:19.254182 23576 ts_manager.cc:194] Registered new tserver with Master: 78e7cc9c944e49d3af0fd3a1c5f4d7c4 (127.7.118.1:35227)
I20260929 18:01:19.258360 23576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.1@KRBTEST.COM'} at 127.7.118.1:37959
I20260929 18:01:19.261686 23764 heartbeater.cc:503] Master 127.7.118.62:45259 was elected leader, sending a full tablet report...
Sep 29 18:01:19 dist-test-slave-r7ph krb5kdc[22994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704878, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for kudu/127.7.118.1@KRBTEST.COM
I20260929 18:01:19.271914 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:19.257666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:35227 (local address 127.0.0.1:33118)
0929 18:01:19.258178 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:19.258238 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:19.258449 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:19.259569 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:01:19.259575 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:01:19.260316 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:01:19.260503 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:19.260512 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.261910 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.261914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:19.262919 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:19.262932 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.263762 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.263768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:19.263907 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:19.264613 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:01:19.264639 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:01:19.266326 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:01:19.269309 (+ 2983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:01:19.269323 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:01:19.269329 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:01:19.269643 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:01:19.270106 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:01:19.270109 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:01:19.270111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:01:19.270228 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:01:19.271052 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:01:19.271058 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:01:19.271344 (+ 286us) client_negotiation.cc:769] Sending connection context
0929 18:01:19.271523 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 18:01:19.271745 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":133,"threads_started":1}
I20260929 18:01:19.272043 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:19.258525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33118 (local address 127.7.118.1:35227)
0929 18:01:19.258881 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:19.258893 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:19.258920 (+ 27us) server_negotiation.cc:407] Connection header received
0929 18:01:19.259005 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:19.259010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:19.259382 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:19.259609 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:01:19.260673 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.261755 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.263081 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.263580 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.263696 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:19.266535 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:01:19.266549 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:01:19.266564 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:01:19.266594 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:01:19.268923 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:01:19.269809 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:01:19.269817 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:01:19.269819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:01:19.269932 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:01:19.270391 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:01:19.270394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:01:19.270396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:01:19.270762 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:01:19.270902 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:19.271660 (+ 758us) server_negotiation.cc:299] Negotiation successful
0929 18:01:19.271862 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":131,"threads_started":1}
I20260929 18:01:19.273186 23698 tablet_service.cc:1514] Processing DeleteTablet for tablet 915571735c7e40afbd5c9d2c1613e9bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:00:58 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:33118
I20260929 18:01:19.274134 23773 tablet_replica.cc:330] stopping tablet replica
I20260929 18:01:19.274501 23773 raft_consensus.cc:2229] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:01:19.274780 23773 raft_consensus.cc:2258] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:01:19.275870 23773 ts_tablet_manager.cc:1905] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:01:19.280320 23773 ts_tablet_manager.cc:1918] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:01:19.280444 23773 log.cc:1196] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/wal/wals/915571735c7e40afbd5c9d2c1613e9bf
I20260929 18:01:19.280820 23773 ts_tablet_manager.cc:1939] T 915571735c7e40afbd5c9d2c1613e9bf P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Deleting consensus metadata
I20260929 18:01:19.283332 23562 catalog_manager.cc:4841] TS 78e7cc9c944e49d3af0fd3a1c5f4d7c4 (127.7.118.1:35227): tablet 915571735c7e40afbd5c9d2c1613e9bf (table test-table [id=a7a9420d5ec943a79a6bae93a8f68ec7]) successfully deleted
W20260929 18:01:19.371415 23768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:01:19.371766 23768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:01:19.371841 23768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:01:19.376222 23768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:01:19.376338 23768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:01:19.376439 23768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 18:01:19.382798 23768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:42109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:34159
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=39091
--tserver_master_addrs=127.7.118.62:45259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:01:19.384016 23768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:01:19.385169 23768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:01:19.396034 23779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:01:19.396114 23778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:01:19.396080 23781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:01:19.396533 23768 server_base.cc:1034] running on GCE node
I20260929 18:01:19.396986 23768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:01:19.397732 23768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:01:19.398939 23768 hybrid_clock.cc:648] HybridClock initialized: now 1790704879398896 us; error 55 us; skew 500 ppm
Sep 29 18:01:19 dist-test-slave-r7ph krb5kdc[22994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704879, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:01:19.402002 23768 init.cc:377] Logged in from keytab as kudu/127.7.118.2@KRBTEST.COM (short username kudu)
I20260929 18:01:19.403003 23768 webserver.cc:460] Webserver started at http://127.7.118.2:39091/ using document root <none> and password file <none>
I20260929 18:01:19.403601 23768 fs_manager.cc:359] Metadata directory not provided
I20260929 18:01:19.403712 23768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:01:19.408052 23768 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 18:01:19.410400 23788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:19.411890 23768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:01:19.412097 23768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "8ba1806660e74869bf5d6276ed729e4f"
format_stamp: "Formatted at 2026-09-29 18:00:58 on dist-test-slave-r7ph"
server_key: "cc99cc47630baed6b31a49ef7b9f6b2f"
server_key_iv: "90e7d25ec8d3598be4adaea6736343ff"
server_key_version: "encryptionkey@0"
I20260929 18:01:19.412701 23768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:01:19.432325 23768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:01:19.436220 23768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:01:19.436439 23768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:01:19.437085 23768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:01:19.437963 23768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:01:19.438081 23768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:19.438153 23768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:01:19.438191 23768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:19.448378 23768 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:34159
I20260929 18:01:19.448398 23901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:34159 every 8 connection(s)
I20260929 18:01:19.449509 23768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 18:01:19.454039 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 23768
I20260929 18:01:19.454584 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:45609
--local_ip_for_outbound_sockets=127.7.118.3
--tserver_master_addrs=127.7.118.62:45259
--webserver_port=40131
--webserver_interface=127.7.118.3
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:42109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:01:19 dist-test-slave-r7ph krb5kdc[22994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704879, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:01:19.464779 23766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:19.451642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:36345 (local address 127.7.118.62:45259)
0929 18:01:19.451816 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:19.451821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:19.452861 (+ 1040us) server_negotiation.cc:407] Connection header received
0929 18:01:19.454197 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:19.454200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:19.454285 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:19.454392 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:01:19.455890 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.456550 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.457860 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.458111 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.458227 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:19.460910 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:01:19.460914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:01:19.460916 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:01:19.460947 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:01:19.462502 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:01:19.463112 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:01:19.463116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:01:19.463118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:01:19.463205 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:01:19.463579 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:01:19.463583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:01:19.463585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:01:19.463793 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:01:19.463899 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:19.464436 (+ 537us) server_negotiation.cc:299] Negotiation successful
0929 18:01:19.464582 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:01:19.465672 23904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:19.452192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.2:36345)
0929 18:01:19.452697 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:19.452744 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:19.453946 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:19.454554 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:01:19.454561 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:01:19.455052 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:01:19.455717 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:19.455734 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.456678 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.456682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:19.457720 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:19.457731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.458318 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.458322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:19.458514 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:19.459161 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:01:19.459181 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:01:19.460705 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:01:19.462662 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:01:19.462669 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:01:19.462690 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:01:19.462995 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:01:19.463319 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:01:19.463322 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:01:19.463324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:01:19.463485 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:01:19.463901 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:01:19.463911 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:01:19.464195 (+ 284us) client_negotiation.cc:769] Sending connection context
0929 18:01:19.464397 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 18:01:19.464761 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":126,"threads_started":1}
I20260929 18:01:19.466964 23902 heartbeater.cc:347] Connected to a master server at 127.7.118.62:45259
I20260929 18:01:19.467248 23902 heartbeater.cc:464] Registering TS with master...
I20260929 18:01:19.467880 23902 heartbeater.cc:511] Master 127.7.118.62:45259 requested a full tablet report, sending...
I20260929 18:01:19.469153 23576 ts_manager.cc:194] Registered new tserver with Master: 8ba1806660e74869bf5d6276ed729e4f (127.7.118.2:34159)
I20260929 18:01:19.469789 23576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.2@KRBTEST.COM'} at 127.7.118.2:36345
W20260929 18:01:19.579353 23905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:01:19.579660 23905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:01:19.579706 23905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:01:19.584079 23905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:01:19.584156 23905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:01:19.584219 23905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 18:01:19.590102 23905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:42109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:45609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=40131
--tserver_master_addrs=127.7.118.62:45259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:01:19.591306 23905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:01:19.592583 23905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:01:19.603283 23910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:01:19.603240 23913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:01:19.603286 23911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:01:19.603482 23905 server_base.cc:1034] running on GCE node
I20260929 18:01:19.604089 23905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:01:19.604853 23905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:01:19.606034 23905 hybrid_clock.cc:648] HybridClock initialized: now 1790704879605999 us; error 58 us; skew 500 ppm
Sep 29 18:01:19 dist-test-slave-r7ph krb5kdc[22994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704879, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:01:19.609107 23905 init.cc:377] Logged in from keytab as kudu/127.7.118.3@KRBTEST.COM (short username kudu)
I20260929 18:01:19.610095 23905 webserver.cc:460] Webserver started at http://127.7.118.3:40131/ using document root <none> and password file <none>
I20260929 18:01:19.610759 23905 fs_manager.cc:359] Metadata directory not provided
I20260929 18:01:19.610865 23905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:01:19.615131 23905 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:01:19.617502 23920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:19.618945 23905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:01:19.619076 23905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "5275fffe40da4606b4e58ec633861a94"
format_stamp: "Formatted at 2026-09-29 18:00:58 on dist-test-slave-r7ph"
server_key: "469e4549a2d26be67494466092c123bf"
server_key_iv: "99a7ccee176155639d6c1c7b4cd3ac76"
server_key_version: "encryptionkey@0"
I20260929 18:01:19.619659 23905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:01:19.636422 23905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:01:19.639974 23905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:01:19.640192 23905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:01:19.640834 23905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:01:19.641783 23905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:01:19.641834 23905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:19.641875 23905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:01:19.641923 23905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:19.651146 23905 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:45609
I20260929 18:01:19.651178 24033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:45609 every 8 connection(s)
I20260929 18:01:19.652796 23905 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 18:01:19.661940 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 23905
Sep 29 18:01:19 dist-test-slave-r7ph krb5kdc[22994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704879, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:01:19.669133 23766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:19.655080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:59945 (local address 127.7.118.62:45259)
0929 18:01:19.655273 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:19.655277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:19.656445 (+ 1168us) server_negotiation.cc:407] Connection header received
0929 18:01:19.657906 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:19.657910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:19.657970 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:19.658232 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:01:19.659797 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.660431 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.661738 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.662108 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.662298 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:19.665217 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:01:19.665220 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:01:19.665223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:01:19.665256 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:01:19.666974 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:01:19.667525 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:01:19.667528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:01:19.667530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:01:19.667617 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:01:19.667991 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:01:19.667995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:01:19.667997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:01:19.668194 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:01:19.668297 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:19.668836 (+ 539us) server_negotiation.cc:299] Negotiation successful
0929 18:01:19.668965 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 18:01:19.670049 24036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:19.655733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.3:59945)
0929 18:01:19.656279 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:19.656326 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:19.657654 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:19.658473 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:01:19.658480 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:01:19.658932 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:01:19.659597 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:19.659615 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.660606 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.660609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:19.661588 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:19.661598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.662417 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.662422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:19.662673 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:19.663384 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:01:19.663402 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:01:19.665030 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:01:19.667094 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:01:19.667101 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:01:19.667119 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:01:19.667418 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:01:19.667725 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:01:19.667728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:01:19.667730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:01:19.667894 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:01:19.668297 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:01:19.668305 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:01:19.668585 (+ 280us) client_negotiation.cc:769] Sending connection context
0929 18:01:19.668794 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 18:01:19.669157 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":182,"threads_started":1}
I20260929 18:01:19.671228 24038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:19.662771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45186 (local address 127.7.118.62:45259)
0929 18:01:19.663045 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:19.663049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:19.663168 (+ 119us) server_negotiation.cc:407] Connection header received
0929 18:01:19.663323 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:19.663326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:19.663379 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:19.663467 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:01:19.664275 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.664908 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.665946 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.666196 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.666332 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:19.667703 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:01:19.667707 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:01:19.667714 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:01:19.667738 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:01:19.669166 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:01:19.669822 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:01:19.669825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:01:19.669827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:01:19.669899 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:01:19.670178 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:01:19.670183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:01:19.670184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:01:19.670484 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:01:19.670710 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:19.670910 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 18:01:19.671072 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":101,"threads_started":1}
I20260929 18:01:19.671509 24034 heartbeater.cc:347] Connected to a master server at 127.7.118.62:45259
I20260929 18:01:19.671808 24034 heartbeater.cc:464] Registering TS with master...
I20260929 18:01:19.672457 24034 heartbeater.cc:511] Master 127.7.118.62:45259 requested a full tablet report, sending...
I20260929 18:01:19.673545 23576 ts_manager.cc:194] Registered new tserver with Master: 5275fffe40da4606b4e58ec633861a94 (127.7.118.3:45609)
I20260929 18:01:19.674113 23576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.3@KRBTEST.COM'} at 127.7.118.3:59945
I20260929 18:01:19.685166 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:01:19.689471 24038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:19.685818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45188 (local address 127.7.118.62:45259)
0929 18:01:19.685970 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:19.685974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:19.686143 (+ 169us) server_negotiation.cc:407] Connection header received
0929 18:01:19.686326 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:19.686330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:19.686454 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:19.686553 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:01:19.686799 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.687345 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.688250 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.688509 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.688653 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:19.688942 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:01:19.689091 (+ 149us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 18:01:19.689312 (+ 221us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45188: token expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 18:01:19.689543 24038 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45188: token expired
W20260929 18:01:19.689896 23501 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.7.118.62:45259)
I20260929 18:01:19.697755 24038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:19.690375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45192 (local address 127.7.118.62:45259)
0929 18:01:19.690501 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:19.690505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:19.690518 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:01:19.690593 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:19.690596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:19.690637 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:19.690723 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:01:19.691327 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.691815 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.692596 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.692818 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.692918 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:19.694377 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:01:19.694380 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:01:19.694383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:01:19.694416 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:01:19.696036 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:01:19.696491 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:01:19.696494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:01:19.696496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:01:19.696581 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:01:19.696883 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:01:19.696886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:01:19.696888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:01:19.697083 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:01:19.697226 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:19.697459 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 18:01:19.697586 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:01:19.700495 23576 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:01:19.703997 23576 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:01:19.713282 23698 tablet_service.cc:1467] Processing CreateTablet for tablet ce7f6feb053640eca50d06719409cee9 (DEFAULT_TABLE table=test-table [id=93cdc4aeec2749ce85bfb0ad74b3f690]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:01:19.714032 23698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce7f6feb053640eca50d06719409cee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:01:19.717474 23656 tablet_bootstrap.cc:492] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Bootstrap starting.
I20260929 18:01:19.718560 23656 tablet_bootstrap.cc:654] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Neither blocks nor log segments found. Creating new log.
I20260929 18:01:19.719770 24044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:19.712447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36028 (local address 127.7.118.3:45609)
0929 18:01:19.712845 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:19.712860 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:19.712887 (+ 27us) server_negotiation.cc:407] Connection header received
0929 18:01:19.712980 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:19.712989 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:19.713256 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:19.713437 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:01:19.713715 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.715505 (+ 1790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.717765 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.718932 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.719101 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:19.719324 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:19.719449 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 18:01:19.719571 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":148,"threads_started":1}
I20260929 18:01:19.719759 24043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:19.712392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:34159 (local address 127.0.0.1:53316)
0929 18:01:19.712931 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:19.712944 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:19.713025 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:19.713973 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:01:19.713976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:01:19.713994 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:01:19.714110 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:19.714118 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.716285 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.716291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:19.717841 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:19.717854 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.719220 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.719224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:19.719339 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:19.719376 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 18:01:19.719436 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 18:01:19.719589 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":28,"thread_start_us":105,"threads_started":1}
I20260929 18:01:19.719792 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:19.712240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:45609 (local address 127.0.0.1:36028)
0929 18:01:19.712389 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:19.712405 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:19.712502 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:19.713433 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:01:19.713436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:01:19.713456 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:01:19.713589 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:19.713596 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.715656 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.715660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:19.717219 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:19.717231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.719104 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.719110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:19.719222 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:19.719307 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 18:01:19.719394 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 18:01:19.719591 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":53}
I20260929 18:01:19.720321 23656 tablet_bootstrap.cc:492] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: No bootstrap required, opened a new log
I20260929 18:01:19.720425 23656 ts_tablet_manager.cc:1397] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:01:19.720896 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:19.712818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53316 (local address 127.7.118.2:34159)
0929 18:01:19.713293 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:19.713305 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:19.713331 (+ 26us) server_negotiation.cc:407] Connection header received
0929 18:01:19.713409 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:19.713415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:19.713835 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:19.714008 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:01:19.714479 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.715839 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.718062 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.719037 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.720167 (+ 1130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:19.720302 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:19.720352 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 18:01:19.720723 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":122,"threads_started":1}
I20260929 18:01:19.720975 23656 raft_consensus.cc:348] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:01:19.721127 23656 raft_consensus.cc:374] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:01:19.721176 23656 raft_consensus.cc:729] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e7cc9c944e49d3af0fd3a1c5f4d7c4, State: Initialized, Role: FOLLOWER
I20260929 18:01:19.721310 23656 consensus_queue.cc:260] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:01:19.721577 23656 ts_tablet_manager.cc:1428] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 18:01:19.724412 23968 tablet_service.cc:1467] Processing CreateTablet for tablet ce7f6feb053640eca50d06719409cee9 (DEFAULT_TABLE table=test-table [id=93cdc4aeec2749ce85bfb0ad74b3f690]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:01:19.724412 23836 tablet_service.cc:1467] Processing CreateTablet for tablet ce7f6feb053640eca50d06719409cee9 (DEFAULT_TABLE table=test-table [id=93cdc4aeec2749ce85bfb0ad74b3f690]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:01:19.726287 23836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce7f6feb053640eca50d06719409cee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:01:19.726291 23968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce7f6feb053640eca50d06719409cee9. 1 dirs total, 0 dirs full, 0 dirs failed
W20260929 18:01:19.728266 23765 tablet.cc:2369] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:01:19.736578 24048 tablet_bootstrap.cc:492] T ce7f6feb053640eca50d06719409cee9 P 5275fffe40da4606b4e58ec633861a94: Bootstrap starting.
I20260929 18:01:19.737468 24047 tablet_bootstrap.cc:492] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f: Bootstrap starting.
I20260929 18:01:19.740186 24048 tablet_bootstrap.cc:654] T ce7f6feb053640eca50d06719409cee9 P 5275fffe40da4606b4e58ec633861a94: Neither blocks nor log segments found. Creating new log.
I20260929 18:01:19.740922 24047 tablet_bootstrap.cc:654] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f: Neither blocks nor log segments found. Creating new log.
I20260929 18:01:19.741225 24048 log.cc:824] T ce7f6feb053640eca50d06719409cee9 P 5275fffe40da4606b4e58ec633861a94: Log is configured to *not* fsync() on all Append() calls
I20260929 18:01:19.742794 24047 log.cc:824] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:01:19.743921 24048 tablet_bootstrap.cc:492] T ce7f6feb053640eca50d06719409cee9 P 5275fffe40da4606b4e58ec633861a94: No bootstrap required, opened a new log
I20260929 18:01:19.744246 24048 ts_tablet_manager.cc:1397] T ce7f6feb053640eca50d06719409cee9 P 5275fffe40da4606b4e58ec633861a94: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260929 18:01:19.745317 24047 tablet_bootstrap.cc:492] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f: No bootstrap required, opened a new log
I20260929 18:01:19.745539 24047 ts_tablet_manager.cc:1397] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 18:01:19.747790 24048 raft_consensus.cc:348] T ce7f6feb053640eca50d06719409cee9 P 5275fffe40da4606b4e58ec633861a94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:01:19.748380 24048 raft_consensus.cc:374] T ce7f6feb053640eca50d06719409cee9 P 5275fffe40da4606b4e58ec633861a94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:01:19.748473 24048 raft_consensus.cc:729] T ce7f6feb053640eca50d06719409cee9 P 5275fffe40da4606b4e58ec633861a94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5275fffe40da4606b4e58ec633861a94, State: Initialized, Role: FOLLOWER
I20260929 18:01:19.749018 24048 consensus_queue.cc:260] T ce7f6feb053640eca50d06719409cee9 P 5275fffe40da4606b4e58ec633861a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:01:19.749843 24034 heartbeater.cc:503] Master 127.7.118.62:45259 was elected leader, sending a full tablet report...
I20260929 18:01:19.750164 24048 ts_tablet_manager.cc:1428] T ce7f6feb053640eca50d06719409cee9 P 5275fffe40da4606b4e58ec633861a94: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260929 18:01:19.750182 24047 raft_consensus.cc:348] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:01:19.750571 24047 raft_consensus.cc:374] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:01:19.750677 24047 raft_consensus.cc:729] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ba1806660e74869bf5d6276ed729e4f, State: Initialized, Role: FOLLOWER
I20260929 18:01:19.751179 24047 consensus_queue.cc:260] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:01:19.752429 23902 heartbeater.cc:503] Master 127.7.118.62:45259 was elected leader, sending a full tablet report...
I20260929 18:01:19.752645 24047 ts_tablet_manager.cc:1428] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260929 18:01:19.774717 24052 raft_consensus.cc:482] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:01:19.775060 24052 raft_consensus.cc:504] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:01:19.776394 24052 leader_election.cc:288] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5275fffe40da4606b4e58ec633861a94 (127.7.118.3:45609), 78e7cc9c944e49d3af0fd3a1c5f4d7c4 (127.7.118.1:35227)
I20260929 18:01:19.780308 23904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:19.776685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:45609 (local address 127.7.118.2:47705)
0929 18:01:19.776799 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:19.776817 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:19.776924 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:19.777245 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:01:19.777249 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:01:19.777271 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:01:19.777406 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:19.777412 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.778150 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.778155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:19.779002 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:19.779012 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.779762 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.779770 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:19.779908 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:19.779933 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:01:19.780061 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 18:01:19.780151 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 18:01:19.780308 24044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:19.776815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:47705 (local address 127.7.118.3:45609)
0929 18:01:19.776947 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:19.776953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:19.776973 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:01:19.777039 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:19.777042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:19.777097 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:19.777220 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:01:19.777527 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.778035 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.779142 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.779564 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.779687 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:19.779842 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:19.780080 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 18:01:19.780173 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:01:19.781193 23988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7f6feb053640eca50d06719409cee9" candidate_uuid: "8ba1806660e74869bf5d6276ed729e4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5275fffe40da4606b4e58ec633861a94" is_pre_election: true
I20260929 18:01:19.781289 24053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:19.776976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:35227 (local address 127.7.118.2:46721)
0929 18:01:19.777467 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:19.777481 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:19.777608 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:19.777890 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:01:19.777893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:01:19.777909 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:01:19.778031 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:19.778036 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.778841 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.778846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:19.779624 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:19.779633 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.780830 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.780834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:19.780943 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:19.780971 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 18:01:19.781050 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 18:01:19.781131 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":171,"threads_started":1}
I20260929 18:01:19.781301 24054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:19.777129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:46721 (local address 127.7.118.1:35227)
0929 18:01:19.777408 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:19.777413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:19.777569 (+ 156us) server_negotiation.cc:407] Connection header received
0929 18:01:19.777709 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:19.777712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:19.777766 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:19.777881 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:01:19.778142 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.778728 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.779753 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.780372 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.780552 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:19.780994 (+ 442us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:19.781071 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 18:01:19.781163 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":64,"threads_started":1}
I20260929 18:01:19.781584 23988 raft_consensus.cc:2454] T ce7f6feb053640eca50d06719409cee9 P 5275fffe40da4606b4e58ec633861a94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ba1806660e74869bf5d6276ed729e4f in term 0.
I20260929 18:01:19.782228 23791 leader_election.cc:302] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5275fffe40da4606b4e58ec633861a94, 8ba1806660e74869bf5d6276ed729e4f; no voters:
I20260929 18:01:19.782582 24052 raft_consensus.cc:2781] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:01:19.782706 24052 raft_consensus.cc:482] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:01:19.782783 24052 raft_consensus.cc:3037] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:01:19.783202 23718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7f6feb053640eca50d06719409cee9" candidate_uuid: "8ba1806660e74869bf5d6276ed729e4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" is_pre_election: true
I20260929 18:01:19.783593 23718 raft_consensus.cc:2454] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ba1806660e74869bf5d6276ed729e4f in term 0.
I20260929 18:01:19.784772 24052 raft_consensus.cc:504] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:01:19.785434 24052 leader_election.cc:288] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f [CANDIDATE]: Term 1 election: Requested vote from peers 5275fffe40da4606b4e58ec633861a94 (127.7.118.3:45609), 78e7cc9c944e49d3af0fd3a1c5f4d7c4 (127.7.118.1:35227)
I20260929 18:01:19.785907 23718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7f6feb053640eca50d06719409cee9" candidate_uuid: "8ba1806660e74869bf5d6276ed729e4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4"
I20260929 18:01:19.786064 23718 raft_consensus.cc:3037] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:01:19.786029 23988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7f6feb053640eca50d06719409cee9" candidate_uuid: "8ba1806660e74869bf5d6276ed729e4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5275fffe40da4606b4e58ec633861a94"
I20260929 18:01:19.786218 23988 raft_consensus.cc:3037] T ce7f6feb053640eca50d06719409cee9 P 5275fffe40da4606b4e58ec633861a94 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:01:19.787284 23718 raft_consensus.cc:2454] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ba1806660e74869bf5d6276ed729e4f in term 1.
I20260929 18:01:19.787663 23791 leader_election.cc:302] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78e7cc9c944e49d3af0fd3a1c5f4d7c4, 8ba1806660e74869bf5d6276ed729e4f; no voters:
I20260929 18:01:19.787755 23988 raft_consensus.cc:2454] T ce7f6feb053640eca50d06719409cee9 P 5275fffe40da4606b4e58ec633861a94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ba1806660e74869bf5d6276ed729e4f in term 1.
I20260929 18:01:19.787932 24052 raft_consensus.cc:2781] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:01:19.788205 24052 raft_consensus.cc:686] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f [term 1 LEADER]: Becoming Leader. State: Replica: 8ba1806660e74869bf5d6276ed729e4f, State: Running, Role: LEADER
I20260929 18:01:19.788584 24052 consensus_queue.cc:237] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:01:19.792290 23576 catalog_manager.cc:5495] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ba1806660e74869bf5d6276ed729e4f (127.7.118.2). New cstate: current_term: 1 leader_uuid: "8ba1806660e74869bf5d6276ed729e4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } health_report { overall_health: UNKNOWN } } }
I20260929 18:01:19.808195 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:19.802733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53332 (local address 127.7.118.2:34159)
0929 18:01:19.802866 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:19.802869 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:19.802884 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:01:19.803214 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:19.803219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:19.803356 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:19.803523 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:01:19.803964 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.804518 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.805817 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:19.806059 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:19.806201 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:19.806729 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:01:19.807355 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:01:19.807472 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:19.807906 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 18:01:19.807981 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:01:19.815305 23718 raft_consensus.cc:1261] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Refusing update from remote peer 8ba1806660e74869bf5d6276ed729e4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:01:19.815311 23988 raft_consensus.cc:1261] T ce7f6feb053640eca50d06719409cee9 P 5275fffe40da4606b4e58ec633861a94 [term 1 FOLLOWER]: Refusing update from remote peer 8ba1806660e74869bf5d6276ed729e4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:01:19.816270 24055 consensus_queue.cc:1035] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:01:19.816643 24052 consensus_queue.cc:1035] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:01:19.825481 24060 mvcc.cc:205] Tried to move back new op lower bound from 7334727187714043904 to 7334727187617206272. Current Snapshot: MvccSnapshot[applied={T|T < 7334727187714043904}]
I20260929 18:01:19.833227 23575 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:01:19.833415 23575 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:01:19.836588 23575 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca: Sending DeleteTablet for 3 replicas of tablet ce7f6feb053640eca50d06719409cee9
I20260929 18:01:19.837515 23698 tablet_service.cc:1514] Processing DeleteTablet for tablet ce7f6feb053640eca50d06719409cee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:01:19 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:33118
I20260929 18:01:19.837560 23836 tablet_service.cc:1514] Processing DeleteTablet for tablet ce7f6feb053640eca50d06719409cee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:01:19 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:53316
I20260929 18:01:19.837828 23968 tablet_service.cc:1514] Processing DeleteTablet for tablet ce7f6feb053640eca50d06719409cee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:01:19 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:36028
I20260929 18:01:19.838135 24069 tablet_replica.cc:330] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f: stopping tablet replica
I20260929 18:01:19.838325 24070 tablet_replica.cc:330] T ce7f6feb053640eca50d06719409cee9 P 5275fffe40da4606b4e58ec633861a94: stopping tablet replica
I20260929 18:01:19.838440 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 23614
I20260929 18:01:19.838534 24069 raft_consensus.cc:2229] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:01:19.839066 24069 raft_consensus.cc:2258] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:01:19.840364 24070 raft_consensus.cc:2229] T ce7f6feb053640eca50d06719409cee9 P 5275fffe40da4606b4e58ec633861a94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:01:19.840727 24070 raft_consensus.cc:2258] T ce7f6feb053640eca50d06719409cee9 P 5275fffe40da4606b4e58ec633861a94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:01:19.840857 24069 ts_tablet_manager.cc:1905] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:01:19.842437 24070 ts_tablet_manager.cc:1905] T ce7f6feb053640eca50d06719409cee9 P 5275fffe40da4606b4e58ec633861a94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:01:19.844007 24069 ts_tablet_manager.cc:1918] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:01:19.844112 24069 log.cc:1196] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/wal/wals/ce7f6feb053640eca50d06719409cee9
I20260929 18:01:19.844447 24069 ts_tablet_manager.cc:1939] T ce7f6feb053640eca50d06719409cee9 P 8ba1806660e74869bf5d6276ed729e4f: Deleting consensus metadata
I20260929 18:01:19.846035 24070 ts_tablet_manager.cc:1918] T ce7f6feb053640eca50d06719409cee9 P 5275fffe40da4606b4e58ec633861a94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:01:19.846146 24070 log.cc:1196] T ce7f6feb053640eca50d06719409cee9 P 5275fffe40da4606b4e58ec633861a94: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/wal/wals/ce7f6feb053640eca50d06719409cee9
I20260929 18:01:19.846218 23561 catalog_manager.cc:4841] TS 8ba1806660e74869bf5d6276ed729e4f (127.7.118.2:34159): tablet ce7f6feb053640eca50d06719409cee9 (table test-table [id=93cdc4aeec2749ce85bfb0ad74b3f690]) successfully deleted
I20260929 18:01:19.846511 24070 ts_tablet_manager.cc:1939] T ce7f6feb053640eca50d06719409cee9 P 5275fffe40da4606b4e58ec633861a94: Deleting consensus metadata
W20260929 18:01:19.847148 23562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.118.1:35227 (error 108)
I20260929 18:01:19.848423 23562 catalog_manager.cc:4841] TS 5275fffe40da4606b4e58ec633861a94 (127.7.118.3:45609): tablet ce7f6feb053640eca50d06719409cee9 (table test-table [id=93cdc4aeec2749ce85bfb0ad74b3f690]) successfully deleted
W20260929 18:01:19.848866 23771 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:01:19.848237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:35227 (local address 127.0.0.1:33132)
0929 18:01:19.848380 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:19.848396 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:19.848494 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:19.848745 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.1:35227: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 18:01:19.849244 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 23768
W20260929 18:01:19.849293 23562 catalog_manager.cc:4584] TS 78e7cc9c944e49d3af0fd3a1c5f4d7c4 (127.7.118.1:35227): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ce7f6feb053640eca50d06719409cee9: Network error: Client connection negotiation failed: client connection to 127.7.118.1:35227: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 18:01:19.855159 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 23905
I20260929 18:01:19.860977 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 23543
I20260929 18:01:39.868373 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:45259
--webserver_interface=127.7.118.62
--webserver_port=36921
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:42109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:45259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:01:39.999406 24072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:01:39.999845 24072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:01:39.999943 24072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:01:40.004593 24072 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 18:01:40.004719 24072 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 18:01:40.004756 24072 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 18:01:40.004791 24072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:01:40.004833 24072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:01:40.004878 24072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:01:40.004915 24072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:01:40.004956 24072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:01:40.011677 24072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:42109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:45259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:45259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=36921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:01:40.013275 24072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:01:40.014762 24072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:01:40.027685 24077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:01:40.027685 24080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:01:40.027731 24078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:01:40.027765 24072 server_base.cc:1034] running on GCE node
I20260929 18:01:40.028839 24072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:01:40.030181 24072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:01:40.031495 24072 hybrid_clock.cc:648] HybridClock initialized: now 1790704900031454 us; error 60 us; skew 500 ppm
Sep 29 18:01:40 dist-test-slave-r7ph krb5kdc[22994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704900, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:01:40.036191 24072 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 18:01:40.037634 24072 webserver.cc:460] Webserver started at http://127.7.118.62:36921/ using document root <none> and password file <none>
I20260929 18:01:40.038312 24072 fs_manager.cc:359] Metadata directory not provided
I20260929 18:01:40.038405 24072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:01:40.043488 24072 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:01:40.046377 24087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:40.048604 24072 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:01:40.048770 24072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "1b69fa140f0a46608927fcf893789cca"
format_stamp: "Formatted at 2026-09-29 18:00:57 on dist-test-slave-r7ph"
server_key: "3e0f36233ea7ad931f835d7c8dacd246"
server_key_iv: "85f6c2a709dfcd0d90204dbde13388bd"
server_key_version: "encryptionkey@0"
I20260929 18:01:40.049609 24072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:01:40.063155 24072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:01:40.067917 24072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:01:40.068125 24072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:01:40.077067 24072 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:45259
I20260929 18:01:40.077384 24139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:45259 every 8 connection(s)
I20260929 18:01:40.078780 24072 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:01:40.084290 24140 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:01:40.086551 24140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca: Bootstrap starting.
I20260929 18:01:40.087409 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 24072
I20260929 18:01:40.087898 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:35227
--local_ip_for_outbound_sockets=127.7.118.1
--tserver_master_addrs=127.7.118.62:45259
--webserver_port=38857
--webserver_interface=127.7.118.1
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:42109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:01:40.107954 24140 log.cc:824] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca: Log is configured to *not* fsync() on all Append() calls
I20260929 18:01:40.121230 24140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:01:40.121695 24140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca: Bootstrap complete.
I20260929 18:01:40.126729 24140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69fa140f0a46608927fcf893789cca" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45259 } }
I20260929 18:01:40.127296 24140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b69fa140f0a46608927fcf893789cca, State: Initialized, Role: FOLLOWER
I20260929 18:01:40.128085 24140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69fa140f0a46608927fcf893789cca" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45259 } }
I20260929 18:01:40.128316 24140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:01:40.128415 24140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:01:40.128561 24140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [term 2 FOLLOWER]: Advancing to term 3
I20260929 18:01:40.131174 24140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69fa140f0a46608927fcf893789cca" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45259 } }
I20260929 18:01:40.131603 24140 leader_election.cc:302] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b69fa140f0a46608927fcf893789cca; no voters:
I20260929 18:01:40.131984 24140 leader_election.cc:288] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 18:01:40.132084 24144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [term 3 FOLLOWER]: Leader election won for term 3
I20260929 18:01:40.132333 24144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [term 3 LEADER]: Becoming Leader. State: Replica: 1b69fa140f0a46608927fcf893789cca, State: Running, Role: LEADER
I20260929 18:01:40.132658 24144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69fa140f0a46608927fcf893789cca" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45259 } }
I20260929 18:01:40.133385 24140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:01:40.134209 24146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b69fa140f0a46608927fcf893789cca. Latest consensus state: current_term: 3 leader_uuid: "1b69fa140f0a46608927fcf893789cca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69fa140f0a46608927fcf893789cca" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45259 } } }
I20260929 18:01:40.134373 24146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [sys.catalog]: This master's current role is: LEADER
I20260929 18:01:40.134294 24145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1b69fa140f0a46608927fcf893789cca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69fa140f0a46608927fcf893789cca" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45259 } } }
I20260929 18:01:40.134444 24145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca [sys.catalog]: This master's current role is: LEADER
I20260929 18:01:40.134778 24153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:01:40.142743 24153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:01:40.143770 24153 catalog_manager.cc:1259] Loaded cluster ID: 217edaa33a804f368501e0b1335b47b2
I20260929 18:01:40.143844 24153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:01:40.146615 24153 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:01:40.147541 24153 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca: Loaded TSK: 0
I20260929 18:01:40.147666 24153 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1b69fa140f0a46608927fcf893789cca: Loaded TSK: 1
F20260929 18:01:40.148557 24153 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc77cc809ad google::LogMessage::Fail() at ??:0
@ 0x7fc77cc8388a google::LogMessage::SendToLog() at ??:0
@ 0x7fc77cc804de google::LogMessage::Flush() at ??:0
@ 0x7fc77cc81a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc77f94418d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc77f93cf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fc7875e57bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fc7875bf99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fc7875c0178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fc787601df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fc78762082b std::function<>::operator()() at ??:0
@ 0x7fc7875bfb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fc7875c0f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc7875bd579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fc787601640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc77eed23d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc77eed2c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fc77eed43f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc77eec3512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc7805076db start_thread at ??:0
@ 0x7fc77b77e61f clone at ??:0
W20260929 18:01:40.241138 24142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:01:40.241508 24142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:01:40.241592 24142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:01:40.246569 24142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:01:40.246625 24142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:01:40.246690 24142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 18:01:40.253047 24142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:42109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:35227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=38857
--tserver_master_addrs=127.7.118.62:45259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:01:40.254374 24142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:01:40.255520 24142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:01:40.266985 24188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:01:40.266953 24190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:01:40.267211 24142 server_base.cc:1034] running on GCE node
W20260929 18:01:40.266985 24187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:01:40.267932 24142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:01:40.268869 24142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:01:40.270097 24142 hybrid_clock.cc:648] HybridClock initialized: now 1790704900270079 us; error 33 us; skew 500 ppm
Sep 29 18:01:40 dist-test-slave-r7ph krb5kdc[22994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704900, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:01:40.274163 24142 init.cc:377] Logged in from keytab as kudu/127.7.118.1@KRBTEST.COM (short username kudu)
I20260929 18:01:40.275638 24142 webserver.cc:460] Webserver started at http://127.7.118.1:38857/ using document root <none> and password file <none>
I20260929 18:01:40.276547 24142 fs_manager.cc:359] Metadata directory not provided
I20260929 18:01:40.276666 24142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:01:40.282573 24142 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 18:01:40.285815 24197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:40.287560 24142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:01:40.287874 24142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4"
format_stamp: "Formatted at 2026-09-29 18:00:57 on dist-test-slave-r7ph"
server_key: "ccc2b037700dec16a4a609473e0c34d5"
server_key_iv: "ff92e57159fbabb0c005879f29a0b813"
server_key_version: "encryptionkey@0"
I20260929 18:01:40.288563 24142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:01:40.305075 24142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:01:40.309355 24142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:01:40.309597 24142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:01:40.310371 24142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:01:40.311650 24204 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:01:40.317802 24142 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:01:40.317947 24142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260929 18:01:40.318059 24142 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:01:40.320195 24142 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:01:40.320309 24142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260929 18:01:40.320632 24204 tablet_bootstrap.cc:492] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Bootstrap starting.
I20260929 18:01:40.332885 24142 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:35227
I20260929 18:01:40.333168 24311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:35227 every 8 connection(s)
I20260929 18:01:40.334059 24142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 18:01:40.337592 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 24142
I20260929 18:01:40.338107 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:34159
--local_ip_for_outbound_sockets=127.7.118.2
--tserver_master_addrs=127.7.118.62:45259
--webserver_port=39091
--webserver_interface=127.7.118.2
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:42109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:01:40.344059 24204 log.cc:824] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Log is configured to *not* fsync() on all Append() calls
I20260929 18:01:40.354831 24204 tablet_bootstrap.cc:492] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:01:40.355396 24204 tablet_bootstrap.cc:492] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Bootstrap complete.
I20260929 18:01:40.356920 24204 ts_tablet_manager.cc:1397] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Time spent bootstrapping tablet: real 0.037s user 0.029s sys 0.004s
I20260929 18:01:40.360760 24204 raft_consensus.cc:348] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:01:40.361392 24204 raft_consensus.cc:729] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e7cc9c944e49d3af0fd3a1c5f4d7c4, State: Initialized, Role: FOLLOWER
I20260929 18:01:40.362016 24204 consensus_queue.cc:260] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:01:40.363307 24204 ts_tablet_manager.cc:1428] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260929 18:01:40.441123 24314 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:01:40.336323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.1:58937)
0929 18:01:40.336861 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:40.336920 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:40.338128 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:40.440142 (+102014us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":174,"threads_started":1}
W20260929 18:01:40.441792 24312 heartbeater.cc:644] Failed to heartbeat to 127.7.118.62:45259 (0 consecutive failures): Network error: Failed to ping master at 127.7.118.62:45259: Client connection negotiation failed: client connection to 127.7.118.62:45259: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 18:01:40.442734 24314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:40.442405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.1:55423)
0929 18:01:40.442571 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:40.442656 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 18:01:40.443593 24314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:40.443352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.1:42279)
0929 18:01:40.443472 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:40.443533 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 18:01:40.443848 24312 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:01:40.476296 24315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:01:40.476601 24315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:01:40.476649 24315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:01:40.481021 24315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:01:40.481104 24315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:01:40.481204 24315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 18:01:40.487213 24315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:42109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:34159
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=39091
--tserver_master_addrs=127.7.118.62:45259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:01:40.488464 24315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:01:40.489663 24315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:01:40.500792 24325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:01:40.500790 24322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:01:40.500794 24323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:01:40.501787 24315 server_base.cc:1034] running on GCE node
I20260929 18:01:40.502393 24315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:01:40.503161 24315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:01:40.504388 24315 hybrid_clock.cc:648] HybridClock initialized: now 1790704900504370 us; error 34 us; skew 500 ppm
Sep 29 18:01:40 dist-test-slave-r7ph krb5kdc[22994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704900, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:01:40.507782 24315 init.cc:377] Logged in from keytab as kudu/127.7.118.2@KRBTEST.COM (short username kudu)
I20260929 18:01:40.508803 24315 webserver.cc:460] Webserver started at http://127.7.118.2:39091/ using document root <none> and password file <none>
I20260929 18:01:40.509433 24315 fs_manager.cc:359] Metadata directory not provided
I20260929 18:01:40.509531 24315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:01:40.514421 24315 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:01:40.517470 24332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:40.519372 24315 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:01:40.519553 24315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "8ba1806660e74869bf5d6276ed729e4f"
format_stamp: "Formatted at 2026-09-29 18:00:58 on dist-test-slave-r7ph"
server_key: "cc99cc47630baed6b31a49ef7b9f6b2f"
server_key_iv: "90e7d25ec8d3598be4adaea6736343ff"
server_key_version: "encryptionkey@0"
I20260929 18:01:40.520222 24315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:01:40.535989 24315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:01:40.540192 24315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:01:40.540485 24315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:01:40.541210 24315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:01:40.542235 24315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:01:40.542471 24315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:40.542622 24315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:01:40.542681 24315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:40.553612 24315 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:34159
I20260929 18:01:40.553665 24445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:34159 every 8 connection(s)
I20260929 18:01:40.554845 24315 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 18:01:40.559763 24448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:40.557760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.2:49493)
0929 18:01:40.558387 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:40.558515 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":138,"threads_started":1}
I20260929 18:01:40.560343 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 24315
W20260929 18:01:40.560385 24446 heartbeater.cc:644] Failed to heartbeat to 127.7.118.62:45259 (0 consecutive failures): Network error: Failed to ping master at 127.7.118.62:45259: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
I20260929 18:01:40.560792 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:45609
--local_ip_for_outbound_sockets=127.7.118.3
--tserver_master_addrs=127.7.118.62:45259
--webserver_port=40131
--webserver_interface=127.7.118.3
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:42109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:01:40.561175 24448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:40.560850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.2:60815)
0929 18:01:40.561028 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:40.561116 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260929 18:01:40.561822 24448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:40.561606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.2:49963)
0929 18:01:40.561714 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:40.561761 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 18:01:40.561991 24446 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:01:40.693404 24449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:01:40.693806 24449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:01:40.693895 24449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:01:40.698915 24449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:01:40.698993 24449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:01:40.699096 24449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 18:01:40.705236 24449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:42109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:45609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=40131
--tserver_master_addrs=127.7.118.62:45259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:01:40.706487 24449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:01:40.707932 24449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:01:40.720224 24457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:01:40.720253 24454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:01:40.720588 24455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:01:40.720715 24449 server_base.cc:1034] running on GCE node
I20260929 18:01:40.721333 24449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:01:40.722527 24449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:01:40.723780 24449 hybrid_clock.cc:648] HybridClock initialized: now 1790704900723717 us; error 82 us; skew 500 ppm
Sep 29 18:01:40 dist-test-slave-r7ph krb5kdc[22994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704900, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:01:40.727221 24449 init.cc:377] Logged in from keytab as kudu/127.7.118.3@KRBTEST.COM (short username kudu)
I20260929 18:01:40.728193 24449 webserver.cc:460] Webserver started at http://127.7.118.3:40131/ using document root <none> and password file <none>
I20260929 18:01:40.728863 24449 fs_manager.cc:359] Metadata directory not provided
I20260929 18:01:40.728932 24449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:01:40.733666 24449 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:01:40.736089 24464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:40.737591 24449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:01:40.737708 24449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "5275fffe40da4606b4e58ec633861a94"
format_stamp: "Formatted at 2026-09-29 18:00:58 on dist-test-slave-r7ph"
server_key: "469e4549a2d26be67494466092c123bf"
server_key_iv: "99a7ccee176155639d6c1c7b4cd3ac76"
server_key_version: "encryptionkey@0"
I20260929 18:01:40.738368 24449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:01:40.764154 24449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:01:40.768355 24449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:01:40.768558 24449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:01:40.769833 24449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:01:40.771003 24449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:01:40.771080 24449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:40.771128 24449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:01:40.771144 24449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:40.782608 24449 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:45609
I20260929 18:01:40.782670 24577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:45609 every 8 connection(s)
I20260929 18:01:40.783931 24449 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 18:01:40.788569 24580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:40.786574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.3:34681)
0929 18:01:40.787151 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:40.787327 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":149,"threads_started":1}
W20260929 18:01:40.789163 24578 heartbeater.cc:644] Failed to heartbeat to 127.7.118.62:45259 (0 consecutive failures): Network error: Failed to ping master at 127.7.118.62:45259: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
I20260929 18:01:40.789881 24580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:40.789640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.3:54553)
0929 18:01:40.789767 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:40.789821 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 18:01:40.789933 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 24449
I20260929 18:01:40.790727 24580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:40.790474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.3:38669)
0929 18:01:40.790586 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:40.790641 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 18:01:40.790907 24578 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 18:01:41.445039 24582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:41.444572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.1:44421)
0929 18:01:41.444862 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:41.444924 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":76,"threads_started":1}
I20260929 18:01:41.563349 24583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:41.562849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.2:51079)
0929 18:01:41.563170 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:41.563236 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":78,"threads_started":1}
I20260929 18:01:41.686946 24584 raft_consensus.cc:482] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:01:41.687222 24584 raft_consensus.cc:504] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:01:41.688549 24584 leader_election.cc:288] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5275fffe40da4606b4e58ec633861a94 (127.7.118.3:45609), 8ba1806660e74869bf5d6276ed729e4f (127.7.118.2:34159)
Sep 29 18:01:41 dist-test-slave-r7ph krb5kdc[22994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704900, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for kudu/127.7.118.3@KRBTEST.COM
Sep 29 18:01:41 dist-test-slave-r7ph krb5kdc[22994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704900, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for kudu/127.7.118.2@KRBTEST.COM
I20260929 18:01:41.709677 24585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:41.688895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:34159 (local address 127.7.118.1:35521)
0929 18:01:41.689359 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:41.689371 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:41.689454 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:41.691384 (+ 1930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:01:41.691386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:01:41.692081 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:01:41.693057 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:41.693082 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:41.695236 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:41.695241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:41.697562 (+ 2321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:41.697581 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:41.698862 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:41.698867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:41.699186 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:41.700060 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:01:41.700087 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:01:41.702822 (+ 2735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:01:41.705682 (+ 2860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:01:41.705691 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:01:41.705710 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:01:41.706485 (+ 775us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:01:41.707212 (+ 727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:01:41.707216 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:01:41.707218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:01:41.707525 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:01:41.708671 (+ 1146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:01:41.708682 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:01:41.709058 (+ 376us) client_negotiation.cc:769] Sending connection context
0929 18:01:41.709241 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 18:01:41.709511 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":586,"spinlock_wait_cycles":33792,"thread_start_us":87,"threads_started":1}
I20260929 18:01:41.709951 24587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:41.689061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:35521 (local address 127.7.118.2:34159)
0929 18:01:41.689432 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:41.689453 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:41.689483 (+ 30us) server_negotiation.cc:407] Connection header received
0929 18:01:41.690348 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:41.690357 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:41.690849 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:41.691221 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:01:41.693294 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:41.694971 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:41.697793 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:41.698686 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:41.698876 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:41.703034 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:01:41.703041 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:01:41.703057 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:01:41.703096 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:01:41.705420 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:01:41.706671 (+ 1251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:01:41.706680 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:01:41.706687 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:01:41.706990 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:01:41.707681 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:01:41.707685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:01:41.707687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:01:41.708411 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:01:41.708603 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:41.709405 (+ 802us) server_negotiation.cc:299] Negotiation successful
0929 18:01:41.709719 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":69,"threads_started":1}
I20260929 18:01:41.710422 24582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:41.688778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:45609 (local address 127.7.118.1:36741)
0929 18:01:41.688978 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:41.688994 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:41.689115 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:41.691342 (+ 2227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:01:41.691350 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:01:41.692081 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:01:41.693057 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:41.693082 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:41.695236 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:41.695241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:41.697562 (+ 2321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:41.697581 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:41.698862 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:41.698867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:41.699176 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:41.700060 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:01:41.700100 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:01:41.702585 (+ 2485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:01:41.705682 (+ 3097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:01:41.705691 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:01:41.705710 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:01:41.706485 (+ 775us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:01:41.707212 (+ 727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:01:41.707216 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:01:41.707218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:01:41.707525 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:01:41.709886 (+ 2361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:01:41.709890 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:01:41.709984 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 18:01:41.710086 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 18:01:41.710276 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":575,"spinlock_wait_cycles":51840}
I20260929 18:01:41.711542 24586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:41.688968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:36741 (local address 127.7.118.3:45609)
0929 18:01:41.689368 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:41.689383 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:41.689428 (+ 45us) server_negotiation.cc:407] Connection header received
0929 18:01:41.690348 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:41.690357 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:41.690849 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:41.691484 (+ 635us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:01:41.693294 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:41.694976 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:41.697793 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:41.698686 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:41.698876 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:41.702848 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:01:41.702856 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:01:41.702898 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:01:41.702942 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:01:41.705420 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:01:41.706693 (+ 1273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:01:41.706700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:01:41.706704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:01:41.706974 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:01:41.707681 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:01:41.707685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:01:41.707687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:01:41.708411 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:01:41.708575 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:41.710752 (+ 2177us) server_negotiation.cc:299] Negotiation successful
0929 18:01:41.711368 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":124,"threads_started":1}
W20260929 18:01:41.713325 24199 leader_election.cc:341] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8ba1806660e74869bf5d6276ed729e4f (127.7.118.2:34159): Not found: Tablet not found: ce7f6feb053640eca50d06719409cee9
W20260929 18:01:41.713963 24200 leader_election.cc:341] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5275fffe40da4606b4e58ec633861a94 (127.7.118.3:45609): Not found: Tablet not found: ce7f6feb053640eca50d06719409cee9
I20260929 18:01:41.714349 24200 leader_election.cc:302] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78e7cc9c944e49d3af0fd3a1c5f4d7c4; no voters: 5275fffe40da4606b4e58ec633861a94, 8ba1806660e74869bf5d6276ed729e4f
I20260929 18:01:41.714731 24584 raft_consensus.cc:2726] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:01:41.792510 24588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:41.791924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.3:49619)
0929 18:01:41.792334 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:41.792399 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":174,"threads_started":1}
I20260929 18:01:42.446593 24589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:42.445998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.1:37591)
0929 18:01:42.446418 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:42.446489 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":103,"threads_started":1}
I20260929 18:01:42.564695 24590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:42.564225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.2:57199)
0929 18:01:42.564520 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:42.564584 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":83,"threads_started":1}
I20260929 18:01:42.794222 24591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:42.793728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.3:34351)
0929 18:01:42.794040 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:42.794107 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":77,"threads_started":1}
I20260929 18:01:43.383280 24592 raft_consensus.cc:482] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:01:43.383497 24592 raft_consensus.cc:504] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:01:43.384210 24592 leader_election.cc:288] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5275fffe40da4606b4e58ec633861a94 (127.7.118.3:45609), 8ba1806660e74869bf5d6276ed729e4f (127.7.118.2:34159)
W20260929 18:01:43.385380 24200 leader_election.cc:341] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5275fffe40da4606b4e58ec633861a94 (127.7.118.3:45609): Not found: Tablet not found: ce7f6feb053640eca50d06719409cee9
W20260929 18:01:43.385567 24199 leader_election.cc:341] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8ba1806660e74869bf5d6276ed729e4f (127.7.118.2:34159): Not found: Tablet not found: ce7f6feb053640eca50d06719409cee9
I20260929 18:01:43.385703 24199 leader_election.cc:302] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78e7cc9c944e49d3af0fd3a1c5f4d7c4; no voters: 5275fffe40da4606b4e58ec633861a94, 8ba1806660e74869bf5d6276ed729e4f
I20260929 18:01:43.386201 24592 raft_consensus.cc:2726] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:01:43.448431 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:43.447825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.1:50423)
0929 18:01:43.448210 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:43.448289 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":79,"threads_started":1}
I20260929 18:01:43.566376 24594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:43.565742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.2:60543)
0929 18:01:43.566119 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:43.566185 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":111,"threads_started":1}
I20260929 18:01:43.795814 24595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:43.795292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.3:38457)
0929 18:01:43.795564 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:43.795648 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260929 18:01:44.449960 24596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:44.449500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.1:39985)
0929 18:01:44.449803 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:44.449863 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":123,"threads_started":1}
I20260929 18:01:44.567865 24597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:44.567345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.2:36555)
0929 18:01:44.567698 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:44.567766 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":84,"threads_started":1}
I20260929 18:01:44.797216 24598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:44.796713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.3:51379)
0929 18:01:44.797044 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:44.797114 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":91,"threads_started":1}
I20260929 18:01:45.348208 24599 raft_consensus.cc:482] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:01:45.348354 24599 raft_consensus.cc:504] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:01:45.348824 24599 leader_election.cc:288] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5275fffe40da4606b4e58ec633861a94 (127.7.118.3:45609), 8ba1806660e74869bf5d6276ed729e4f (127.7.118.2:34159)
W20260929 18:01:45.349880 24199 leader_election.cc:341] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8ba1806660e74869bf5d6276ed729e4f (127.7.118.2:34159): Not found: Tablet not found: ce7f6feb053640eca50d06719409cee9
W20260929 18:01:45.349985 24200 leader_election.cc:341] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5275fffe40da4606b4e58ec633861a94 (127.7.118.3:45609): Not found: Tablet not found: ce7f6feb053640eca50d06719409cee9
I20260929 18:01:45.350035 24199 leader_election.cc:302] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78e7cc9c944e49d3af0fd3a1c5f4d7c4; no voters: 5275fffe40da4606b4e58ec633861a94, 8ba1806660e74869bf5d6276ed729e4f
I20260929 18:01:45.350172 24599 raft_consensus.cc:2726] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:01:45.451619 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:45.451101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.1:55337)
0929 18:01:45.451450 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:45.451520 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":86,"threads_started":1}
I20260929 18:01:45.569432 24601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:45.568909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.2:46863)
0929 18:01:45.569247 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:45.569312 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":90,"threads_started":1}
I20260929 18:01:45.798710 24602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:45.798184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.3:36245)
0929 18:01:45.798529 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:45.798608 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":170,"threads_started":1}
I20260929 18:01:46.452890 24603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:46.452457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.1:34537)
0929 18:01:46.452731 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:46.452793 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":81,"threads_started":1}
I20260929 18:01:46.570920 24604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:46.570478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.2:37019)
0929 18:01:46.570751 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:46.570823 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":99,"threads_started":1}
I20260929 18:01:46.800040 24605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:46.799544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.3:59895)
0929 18:01:46.799855 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:46.799927 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":79,"threads_started":1}
I20260929 18:01:47.454222 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:47.453757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.1:36123)
0929 18:01:47.454054 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:47.454127 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":91,"threads_started":1}
I20260929 18:01:47.572345 24607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:47.571841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.2:43777)
0929 18:01:47.572181 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:47.572245 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":79,"threads_started":1}
I20260929 18:01:47.801542 24608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:47.800989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.3:60791)
0929 18:01:47.801348 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:47.801417 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":76,"threads_started":1}
I20260929 18:01:48.455646 24609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:48.455198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.1:38295)
0929 18:01:48.455482 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:48.455547 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":87,"threads_started":1}
I20260929 18:01:48.573702 24610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:48.573233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.2:40221)
0929 18:01:48.573534 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:48.573594 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":81,"threads_started":1}
I20260929 18:01:48.803113 24611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:48.802594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.3:57233)
0929 18:01:48.802939 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:48.803012 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":89,"threads_started":1}
I20260929 18:01:49.457018 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:49.456559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.1:59629)
0929 18:01:49.456855 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:49.456921 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":81,"threads_started":1}
I20260929 18:01:49.575403 24613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:49.574848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.2:42505)
0929 18:01:49.575170 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:49.575281 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":79,"threads_started":1}
I20260929 18:01:49.681259 24614 raft_consensus.cc:482] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:01:49.681445 24614 raft_consensus.cc:504] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:01:49.681969 24614 leader_election.cc:288] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5275fffe40da4606b4e58ec633861a94 (127.7.118.3:45609), 8ba1806660e74869bf5d6276ed729e4f (127.7.118.2:34159)
W20260929 18:01:49.682811 24199 leader_election.cc:341] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8ba1806660e74869bf5d6276ed729e4f (127.7.118.2:34159): Not found: Tablet not found: ce7f6feb053640eca50d06719409cee9
W20260929 18:01:49.682906 24200 leader_election.cc:341] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5275fffe40da4606b4e58ec633861a94 (127.7.118.3:45609): Not found: Tablet not found: ce7f6feb053640eca50d06719409cee9
I20260929 18:01:49.682956 24200 leader_election.cc:302] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78e7cc9c944e49d3af0fd3a1c5f4d7c4; no voters: 5275fffe40da4606b4e58ec633861a94, 8ba1806660e74869bf5d6276ed729e4f
I20260929 18:01:49.683107 24614 raft_consensus.cc:2726] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:01:49.804569 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:49.804086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.3:37631)
0929 18:01:49.804402 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:49.804469 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":81,"threads_started":1}
I20260929 18:01:50.458488 24616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:50.457980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.1:33029)
0929 18:01:50.458304 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:50.458375 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":122,"threads_started":1}
I20260929 18:01:50.576895 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:50.576417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.2:44225)
0929 18:01:50.576697 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:50.576763 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":80,"threads_started":1}
I20260929 18:01:50.805958 24618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:50.805469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.3:40573)
0929 18:01:50.805781 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:50.805846 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":95,"threads_started":1}
I20260929 18:01:51.459847 24619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:51.459394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.1:54513)
0929 18:01:51.459681 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:51.459746 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":88,"threads_started":1}
I20260929 18:01:51.578377 24620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:51.577806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.2:34999)
0929 18:01:51.578115 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:51.578182 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":81,"threads_started":1}
I20260929 18:01:51.807456 24621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:51.806844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.3:58509)
0929 18:01:51.807239 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:51.807312 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":111,"threads_started":1}
I20260929 18:01:52.443233 24622 raft_consensus.cc:482] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:01:52.443382 24622 raft_consensus.cc:504] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:01:52.443931 24622 leader_election.cc:288] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5275fffe40da4606b4e58ec633861a94 (127.7.118.3:45609), 8ba1806660e74869bf5d6276ed729e4f (127.7.118.2:34159)
W20260929 18:01:52.445017 24199 leader_election.cc:341] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8ba1806660e74869bf5d6276ed729e4f (127.7.118.2:34159): Not found: Tablet not found: ce7f6feb053640eca50d06719409cee9
W20260929 18:01:52.445194 24200 leader_election.cc:341] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5275fffe40da4606b4e58ec633861a94 (127.7.118.3:45609): Not found: Tablet not found: ce7f6feb053640eca50d06719409cee9
I20260929 18:01:52.445303 24200 leader_election.cc:302] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78e7cc9c944e49d3af0fd3a1c5f4d7c4; no voters: 5275fffe40da4606b4e58ec633861a94, 8ba1806660e74869bf5d6276ed729e4f
I20260929 18:01:52.445503 24622 raft_consensus.cc:2726] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:01:52.461489 24623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:52.460979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.1:43375)
0929 18:01:52.461323 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:52.461390 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":98,"threads_started":1}
I20260929 18:01:52.579919 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:52.579401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.2:38257)
0929 18:01:52.579753 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:52.579816 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":92,"threads_started":1}
I20260929 18:01:52.808987 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:52.808497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.3:36127)
0929 18:01:52.808821 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:52.808887 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":121,"threads_started":1}
I20260929 18:01:53.462953 24626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:53.462502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.1:59613)
0929 18:01:53.462781 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:53.462853 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":99,"threads_started":1}
I20260929 18:01:53.581408 24627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:53.580895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.2:53457)
0929 18:01:53.581232 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:53.581308 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":118,"threads_started":1}
I20260929 18:01:53.810654 24628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:53.810066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.3:57295)
0929 18:01:53.810457 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:53.810531 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":102,"threads_started":1}
I20260929 18:01:54.464468 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:54.463973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.1:45875)
0929 18:01:54.464306 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:54.464370 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":124,"threads_started":1}
I20260929 18:01:54.547194 24630 raft_consensus.cc:482] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:01:54.547384 24630 raft_consensus.cc:504] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5275fffe40da4606b4e58ec633861a94" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 45609 } } peers { permanent_uuid: "8ba1806660e74869bf5d6276ed729e4f" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34159 } } peers { permanent_uuid: "78e7cc9c944e49d3af0fd3a1c5f4d7c4" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35227 } }
I20260929 18:01:54.547897 24630 leader_election.cc:288] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5275fffe40da4606b4e58ec633861a94 (127.7.118.3:45609), 8ba1806660e74869bf5d6276ed729e4f (127.7.118.2:34159)
W20260929 18:01:54.548806 24200 leader_election.cc:341] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5275fffe40da4606b4e58ec633861a94 (127.7.118.3:45609): Not found: Tablet not found: ce7f6feb053640eca50d06719409cee9
W20260929 18:01:54.548919 24199 leader_election.cc:341] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8ba1806660e74869bf5d6276ed729e4f (127.7.118.2:34159): Not found: Tablet not found: ce7f6feb053640eca50d06719409cee9
I20260929 18:01:54.548980 24199 leader_election.cc:302] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78e7cc9c944e49d3af0fd3a1c5f4d7c4; no voters: 5275fffe40da4606b4e58ec633861a94, 8ba1806660e74869bf5d6276ed729e4f
I20260929 18:01:54.549119 24630 raft_consensus.cc:2726] T ce7f6feb053640eca50d06719409cee9 P 78e7cc9c944e49d3af0fd3a1c5f4d7c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:01:54.582940 24631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:54.582444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.2:56979)
0929 18:01:54.582775 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:54.582842 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":78,"threads_started":1}
I20260929 18:01:54.812062 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:54.811542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.3:39397)
0929 18:01:54.811885 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:54.811948 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":96,"threads_started":1}
I20260929 18:01:55.466037 24633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:55.465572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.1:36285)
0929 18:01:55.465880 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:55.465940 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":89,"threads_started":1}
I20260929 18:01:55.584693 24634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:55.584007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45259 (local address 127.7.118.2:57489)
0929 18:01:55.584499 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:55.584566 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.62:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":105,"threads_started":1}
I20260929 18:01:55.791227 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 24142
I20260929 18:01:55.797063 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 24315
I20260929 18:01:55.802935 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 24449
2026-09-29T18:01:55Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59620 ms)
[----------] 30 tests from SecurityITest (258229 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:01:55 dist-test-slave-r7ph krb5kdc[24637](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:01:55 dist-test-slave-r7ph krb5kdc[24637](info): set up 2 sockets
Sep 29 18:01:55 dist-test-slave-r7ph krb5kdc[24637](info): commencing operation
krb5kdc: starting...
W20260929 18:01:56.862315 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 18:01:56 dist-test-slave-r7ph krb5kdc[24637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:01:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:01:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 18:01:56.994354 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:40583
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:40583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:01:57.115415 24650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:01:57.115797 24650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:01:57.115903 24650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:01:57.120997 24650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:01:57.121107 24650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:01:57.121148 24650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:01:57.121212 24650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:01:57.121270 24650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:01:57.127811 24650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:40583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:40583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:01:57.129091 24650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:01:57.130241 24650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:01:57.140437 24650 server_base.cc:1034] running on GCE node
W20260929 18:01:57.140357 24658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:01:57.140525 24655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:01:57.140785 24656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:01:57.141645 24650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:01:57.143101 24650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:01:57.144352 24650 hybrid_clock.cc:648] HybridClock initialized: now 1790704917144311 us; error 62 us; skew 500 ppm
Sep 29 18:01:57 dist-test-slave-r7ph krb5kdc[24637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704917, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:01:57.147933 24650 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 18:01:57.149082 24650 webserver.cc:460] Webserver started at http://127.7.118.62:39289/ using document root <none> and password file <none>
I20260929 18:01:57.149694 24650 fs_manager.cc:359] Metadata directory not provided
I20260929 18:01:57.149762 24650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:01:57.149953 24650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:01:57.152515 24650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "b47e515197af444891a7f0e979048bb7"
format_stamp: "Formatted at 2026-09-29 18:01:57 on dist-test-slave-r7ph"
server_key: "7d639b3b4383fd290b36ab67518b2c1d"
server_key_iv: "57983ea4bee139a2bce7c9c39fae283b"
server_key_version: "encryptionkey@0"
I20260929 18:01:57.153126 24650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "b47e515197af444891a7f0e979048bb7"
format_stamp: "Formatted at 2026-09-29 18:01:57 on dist-test-slave-r7ph"
server_key: "7d639b3b4383fd290b36ab67518b2c1d"
server_key_iv: "57983ea4bee139a2bce7c9c39fae283b"
server_key_version: "encryptionkey@0"
I20260929 18:01:57.157637 24650 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:01:57.160600 24665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:57.162077 24650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:01:57.162241 24650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "b47e515197af444891a7f0e979048bb7"
format_stamp: "Formatted at 2026-09-29 18:01:57 on dist-test-slave-r7ph"
server_key: "7d639b3b4383fd290b36ab67518b2c1d"
server_key_iv: "57983ea4bee139a2bce7c9c39fae283b"
server_key_version: "encryptionkey@0"
I20260929 18:01:57.162434 24650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:01:57.186241 24650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:01:57.189924 24650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:01:57.190162 24650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:01:57.197887 24650 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:40583
I20260929 18:01:57.197916 24717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:40583 every 8 connection(s)
I20260929 18:01:57.198949 24650 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:01:57.201726 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 24650
I20260929 18:01:57.201848 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:01:57.202061 24718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:01:57.202234 7640 external_mini_cluster.cc:1428] Setting key 5749b11169a9d703211c814d7ba10637
I20260929 18:01:57.209044 24718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b47e515197af444891a7f0e979048bb7: Bootstrap starting.
I20260929 18:01:57.212005 24718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b47e515197af444891a7f0e979048bb7: Neither blocks nor log segments found. Creating new log.
Sep 29 18:01:57 dist-test-slave-r7ph krb5kdc[24637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:01:57.212945 24718 log.cc:824] T 00000000000000000000000000000000 P b47e515197af444891a7f0e979048bb7: Log is configured to *not* fsync() on all Append() calls
I20260929 18:01:57.215476 24718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b47e515197af444891a7f0e979048bb7: No bootstrap required, opened a new log
I20260929 18:01:57.218407 24721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:57.203779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47034 (local address 127.7.118.62:40583)
0929 18:01:57.204223 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:57.204238 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:57.204275 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:01:57.205228 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:57.205249 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:57.205645 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:57.205943 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:01:57.206926 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:57.207928 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:57.209100 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:57.209683 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:57.209816 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:57.212756 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:01:57.212762 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:01:57.212778 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:01:57.212809 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:01:57.214879 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:01:57.215363 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:01:57.215369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:01:57.215375 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:01:57.215526 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:01:57.215867 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:01:57.215872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:01:57.215874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:01:57.216410 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:01:57.216608 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:57.216893 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 18:01:57.217227 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":153,"threads_started":1}
I20260929 18:01:57.219961 24718 raft_consensus.cc:348] T 00000000000000000000000000000000 P b47e515197af444891a7f0e979048bb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47e515197af444891a7f0e979048bb7" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 40583 } }
I20260929 18:01:57.220207 24718 raft_consensus.cc:374] T 00000000000000000000000000000000 P b47e515197af444891a7f0e979048bb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:01:57.220281 24718 raft_consensus.cc:729] T 00000000000000000000000000000000 P b47e515197af444891a7f0e979048bb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b47e515197af444891a7f0e979048bb7, State: Initialized, Role: FOLLOWER
I20260929 18:01:57.220716 24718 consensus_queue.cc:260] T 00000000000000000000000000000000 P b47e515197af444891a7f0e979048bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47e515197af444891a7f0e979048bb7" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 40583 } }
I20260929 18:01:57.220890 24718 raft_consensus.cc:388] T 00000000000000000000000000000000 P b47e515197af444891a7f0e979048bb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:01:57.220990 24718 raft_consensus.cc:482] T 00000000000000000000000000000000 P b47e515197af444891a7f0e979048bb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:01:57.221097 24718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b47e515197af444891a7f0e979048bb7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:01:57.222039 24718 raft_consensus.cc:504] T 00000000000000000000000000000000 P b47e515197af444891a7f0e979048bb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47e515197af444891a7f0e979048bb7" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 40583 } }
I20260929 18:01:57.222472 24718 leader_election.cc:302] T 00000000000000000000000000000000 P b47e515197af444891a7f0e979048bb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b47e515197af444891a7f0e979048bb7; no voters:
I20260929 18:01:57.222858 24718 leader_election.cc:288] T 00000000000000000000000000000000 P b47e515197af444891a7f0e979048bb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:01:57.223045 24723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b47e515197af444891a7f0e979048bb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:01:57.223313 24723 raft_consensus.cc:686] T 00000000000000000000000000000000 P b47e515197af444891a7f0e979048bb7 [term 1 LEADER]: Becoming Leader. State: Replica: b47e515197af444891a7f0e979048bb7, State: Running, Role: LEADER
I20260929 18:01:57.223744 24723 consensus_queue.cc:237] T 00000000000000000000000000000000 P b47e515197af444891a7f0e979048bb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47e515197af444891a7f0e979048bb7" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 40583 } }
I20260929 18:01:57.224299 24718 sys_catalog.cc:564] T 00000000000000000000000000000000 P b47e515197af444891a7f0e979048bb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:01:57.225628 24724 sys_catalog.cc:455] T 00000000000000000000000000000000 P b47e515197af444891a7f0e979048bb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b47e515197af444891a7f0e979048bb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47e515197af444891a7f0e979048bb7" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 40583 } } }
I20260929 18:01:57.225703 24725 sys_catalog.cc:455] T 00000000000000000000000000000000 P b47e515197af444891a7f0e979048bb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b47e515197af444891a7f0e979048bb7. Latest consensus state: current_term: 1 leader_uuid: "b47e515197af444891a7f0e979048bb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47e515197af444891a7f0e979048bb7" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 40583 } } }
I20260929 18:01:57.225778 24724 sys_catalog.cc:458] T 00000000000000000000000000000000 P b47e515197af444891a7f0e979048bb7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:01:57.225831 24725 sys_catalog.cc:458] T 00000000000000000000000000000000 P b47e515197af444891a7f0e979048bb7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:01:57.226179 24732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:01:57.229707 24732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:01:57.236145 24732 catalog_manager.cc:1347] Generated new cluster ID: d69c82781db943f9bcfa1f6474c62852
I20260929 18:01:57.236234 24732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:01:57.255275 24732 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:01:57.256750 24732 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:01:57.268453 24732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b47e515197af444891a7f0e979048bb7: Generated new TSK 0
I20260929 18:01:57.269449 24732 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.118.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
I20260929 18:01:57.306942 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:0
--local_ip_for_outbound_sockets=127.7.118.1
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:40583
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:01:57.429004 24744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:01:57.429314 24744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:01:57.429360 24744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:01:57.433804 24744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:01:57.433888 24744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:01:57.433954 24744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 18:01:57.439729 24744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:40583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:01:57.440994 24744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:01:57.442181 24744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:01:57.452696 24750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:01:57.452702 24752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:01:57.452829 24749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:01:57.453092 24744 server_base.cc:1034] running on GCE node
I20260929 18:01:57.453555 24744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:01:57.454311 24744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:01:57.455528 24744 hybrid_clock.cc:648] HybridClock initialized: now 1790704917455511 us; error 33 us; skew 500 ppm
Sep 29 18:01:57 dist-test-slave-r7ph krb5kdc[24637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704917, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:01:57.458590 24744 init.cc:377] Logged in from keytab as kudu/127.7.118.1@KRBTEST.COM (short username kudu)
I20260929 18:01:57.459570 24744 webserver.cc:460] Webserver started at http://127.7.118.1:39345/ using document root <none> and password file <none>
I20260929 18:01:57.460206 24744 fs_manager.cc:359] Metadata directory not provided
I20260929 18:01:57.460306 24744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:01:57.460565 24744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:01:57.462886 24744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-0/data/instance:
uuid: "c61f5856655948a79b1d60f6db3e2ee3"
format_stamp: "Formatted at 2026-09-29 18:01:57 on dist-test-slave-r7ph"
server_key: "f70998d7b74800ec5a8edbc88310f204"
server_key_iv: "9c85372c045f51ffd28efbac6f5f5a89"
server_key_version: "encryptionkey@0"
I20260929 18:01:57.463493 24744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance:
uuid: "c61f5856655948a79b1d60f6db3e2ee3"
format_stamp: "Formatted at 2026-09-29 18:01:57 on dist-test-slave-r7ph"
server_key: "f70998d7b74800ec5a8edbc88310f204"
server_key_iv: "9c85372c045f51ffd28efbac6f5f5a89"
server_key_version: "encryptionkey@0"
I20260929 18:01:57.467744 24744 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 18:01:57.471130 24759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:57.472975 24744 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:01:57.473125 24744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "c61f5856655948a79b1d60f6db3e2ee3"
format_stamp: "Formatted at 2026-09-29 18:01:57 on dist-test-slave-r7ph"
server_key: "f70998d7b74800ec5a8edbc88310f204"
server_key_iv: "9c85372c045f51ffd28efbac6f5f5a89"
server_key_version: "encryptionkey@0"
I20260929 18:01:57.473312 24744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:01:57.486174 24744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:01:57.489538 24744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:01:57.489771 24744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:01:57.490471 24744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:01:57.491341 24744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:01:57.491415 24744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:57.491488 24744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:01:57.491542 24744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:57.502665 24744 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:33795
I20260929 18:01:57.502686 24872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:33795 every 8 connection(s)
I20260929 18:01:57.503639 24744 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 18:01:57.505030 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 24744
I20260929 18:01:57.505180 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance
I20260929 18:01:57.505575 7640 external_mini_cluster.cc:1428] Setting key dd23b2fd9d622ac670a4f1e2a93ad82e
Sep 29 18:01:57 dist-test-slave-r7ph krb5kdc[24637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704917, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.118.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.2@KRBTEST.COM" created.
I20260929 18:01:57.520550 24721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:57.505789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:51631 (local address 127.7.118.62:40583)
0929 18:01:57.505923 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:57.505926 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:57.507031 (+ 1105us) server_negotiation.cc:407] Connection header received
0929 18:01:57.508394 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:57.508397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:57.508446 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:57.508526 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:01:57.509988 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:57.510912 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:57.512159 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:57.512525 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:57.512624 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:57.516140 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:01:57.516144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:01:57.516146 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:01:57.516174 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:01:57.517726 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:01:57.518387 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:01:57.518391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:01:57.518393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:01:57.518510 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:01:57.518931 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:01:57.518934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:01:57.518935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:01:57.519258 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:01:57.519452 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:57.520129 (+ 677us) server_negotiation.cc:299] Negotiation successful
0929 18:01:57.520300 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:01:57.521312 24875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:57.506300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:40583 (local address 127.7.118.1:51631)
0929 18:01:57.506857 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:57.506922 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:57.508080 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:57.508750 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:01:57.508763 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:01:57.509314 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:01:57.509829 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:57.509843 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:57.511084 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:57.511087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:57.512027 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:57.512039 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:57.512741 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:57.512748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:57.513061 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:57.513956 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:01:57.513976 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:01:57.515881 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:01:57.517862 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:01:57.517869 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:01:57.517894 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:01:57.518203 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:01:57.518619 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:01:57.518622 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:01:57.518623 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:01:57.518798 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:01:57.519471 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:01:57.519482 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:01:57.519842 (+ 360us) client_negotiation.cc:769] Sending connection context
0929 18:01:57.520021 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 18:01:57.520321 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":122,"threads_started":1}
I20260929 18:01:57.522792 24873 heartbeater.cc:347] Connected to a master server at 127.7.118.62:40583
I20260929 18:01:57.523085 24873 heartbeater.cc:464] Registering TS with master...
I20260929 18:01:57.523747 24873 heartbeater.cc:511] Master 127.7.118.62:40583 requested a full tablet report, sending...
I20260929 18:01:57.525647 24682 ts_manager.cc:194] Registered new tserver with Master: c61f5856655948a79b1d60f6db3e2ee3 (127.7.118.1:33795)
I20260929 18:01:57.527024 24682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.1@KRBTEST.COM'} at 127.7.118.1:51631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
I20260929 18:01:57.539050 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:0
--local_ip_for_outbound_sockets=127.7.118.2
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:40583
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:01:57.660043 24878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:01:57.660382 24878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:01:57.660444 24878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:01:57.665220 24878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:01:57.665326 24878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:01:57.665437 24878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 18:01:57.671612 24878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:40583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:01:57.672873 24878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:01:57.673974 24878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:01:57.684909 24883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:01:57.685093 24878 server_base.cc:1034] running on GCE node
W20260929 18:01:57.684993 24886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:01:57.684928 24884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:01:57.685845 24878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:01:57.686646 24878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:01:57.687853 24878 hybrid_clock.cc:648] HybridClock initialized: now 1790704917687828 us; error 45 us; skew 500 ppm
Sep 29 18:01:57 dist-test-slave-r7ph krb5kdc[24637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704917, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:01:57.690973 24878 init.cc:377] Logged in from keytab as kudu/127.7.118.2@KRBTEST.COM (short username kudu)
I20260929 18:01:57.691913 24878 webserver.cc:460] Webserver started at http://127.7.118.2:40573/ using document root <none> and password file <none>
I20260929 18:01:57.692466 24878 fs_manager.cc:359] Metadata directory not provided
I20260929 18:01:57.692526 24878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:01:57.692808 24878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:01:57.695108 24878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-1/data/instance:
uuid: "4526b6aced19411d8fc19800b5dfb8f3"
format_stamp: "Formatted at 2026-09-29 18:01:57 on dist-test-slave-r7ph"
server_key: "1aecb2b54c27f8a539ccfa8475262129"
server_key_iv: "7e3dc8b55d544d0b4f842085c102f981"
server_key_version: "encryptionkey@0"
I20260929 18:01:57.695704 24878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance:
uuid: "4526b6aced19411d8fc19800b5dfb8f3"
format_stamp: "Formatted at 2026-09-29 18:01:57 on dist-test-slave-r7ph"
server_key: "1aecb2b54c27f8a539ccfa8475262129"
server_key_iv: "7e3dc8b55d544d0b4f842085c102f981"
server_key_version: "encryptionkey@0"
I20260929 18:01:57.699692 24878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:01:57.702520 24893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:57.703970 24878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:01:57.704107 24878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "4526b6aced19411d8fc19800b5dfb8f3"
format_stamp: "Formatted at 2026-09-29 18:01:57 on dist-test-slave-r7ph"
server_key: "1aecb2b54c27f8a539ccfa8475262129"
server_key_iv: "7e3dc8b55d544d0b4f842085c102f981"
server_key_version: "encryptionkey@0"
I20260929 18:01:57.704218 24878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:01:57.728659 24878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:01:57.732435 24878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:01:57.732640 24878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:01:57.733210 24878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:01:57.734054 24878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:01:57.734104 24878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:57.734146 24878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:01:57.734161 24878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:57.743640 24878 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:46193
I20260929 18:01:57.743659 25006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:46193 every 8 connection(s)
I20260929 18:01:57.744612 24878 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 18:01:57.746852 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 24878
I20260929 18:01:57.747025 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance
I20260929 18:01:57.747318 7640 external_mini_cluster.cc:1428] Setting key 30c6989f660dd28f13e6d0ae5f0c0b03
Sep 29 18:01:57 dist-test-slave-r7ph krb5kdc[24637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704917, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.118.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.3@KRBTEST.COM" created.
I20260929 18:01:57.760030 24721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:57.746643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:41855 (local address 127.7.118.62:40583)
0929 18:01:57.746827 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:57.746833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:57.747845 (+ 1012us) server_negotiation.cc:407] Connection header received
0929 18:01:57.749038 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:57.749041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:57.749092 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:57.749172 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:01:57.750962 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:57.751499 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:57.752697 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:57.752938 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:57.753063 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:57.755898 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:01:57.755902 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:01:57.755904 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:01:57.755933 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:01:57.757502 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:01:57.758222 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:01:57.758225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:01:57.758227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:01:57.758356 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:01:57.758730 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:01:57.758735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:01:57.758737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:01:57.758972 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:01:57.759126 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:57.759717 (+ 591us) server_negotiation.cc:299] Negotiation successful
0929 18:01:57.759863 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 18:01:57.761210 25009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:57.747060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:40583 (local address 127.7.118.2:41855)
0929 18:01:57.747634 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:57.747711 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:57.748741 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:57.749404 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:01:57.749417 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:01:57.749982 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:01:57.750803 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:57.750821 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:57.751639 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:57.751642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:57.752559 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:57.752569 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:57.753106 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:57.753111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:57.753302 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:57.753995 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:01:57.754011 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:01:57.755692 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:01:57.757702 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:01:57.757708 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:01:57.757722 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:01:57.758104 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:01:57.758451 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:01:57.758455 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:01:57.758457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:01:57.758630 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:01:57.759123 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:01:57.759131 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:01:57.759425 (+ 294us) client_negotiation.cc:769] Sending connection context
0929 18:01:57.759660 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 18:01:57.760034 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":149,"threads_started":1}
I20260929 18:01:57.762714 25007 heartbeater.cc:347] Connected to a master server at 127.7.118.62:40583
I20260929 18:01:57.763072 25007 heartbeater.cc:464] Registering TS with master...
I20260929 18:01:57.763756 25007 heartbeater.cc:511] Master 127.7.118.62:40583 requested a full tablet report, sending...
I20260929 18:01:57.765251 24682 ts_manager.cc:194] Registered new tserver with Master: 4526b6aced19411d8fc19800b5dfb8f3 (127.7.118.2:46193)
I20260929 18:01:57.766314 24682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.2@KRBTEST.COM'} at 127.7.118.2:41855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
I20260929 18:01:57.779932 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:0
--local_ip_for_outbound_sockets=127.7.118.3
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:40583
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:01:57.904215 25012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:01:57.904546 25012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:01:57.904593 25012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:01:57.909143 25012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:01:57.909220 25012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:01:57.909287 25012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 18:01:57.915045 25012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:40583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:01:57.916206 25012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:01:57.917376 25012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:01:57.927714 25020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:01:57.927714 25017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:01:57.927735 25018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:01:57.927954 25012 server_base.cc:1034] running on GCE node
I20260929 18:01:57.928465 25012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:01:57.929198 25012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:01:57.930397 25012 hybrid_clock.cc:648] HybridClock initialized: now 1790704917930342 us; error 67 us; skew 500 ppm
Sep 29 18:01:57 dist-test-slave-r7ph krb5kdc[24637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704917, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:01:57.933403 25012 init.cc:377] Logged in from keytab as kudu/127.7.118.3@KRBTEST.COM (short username kudu)
I20260929 18:01:57.934515 25012 webserver.cc:460] Webserver started at http://127.7.118.3:34905/ using document root <none> and password file <none>
I20260929 18:01:57.935098 25012 fs_manager.cc:359] Metadata directory not provided
I20260929 18:01:57.935165 25012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:01:57.935340 25012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:01:57.937624 25012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-2/data/instance:
uuid: "bdc774ac71a847c4853bd7b3cbbd620b"
format_stamp: "Formatted at 2026-09-29 18:01:57 on dist-test-slave-r7ph"
server_key: "e5f5491f9bbfe3ff00a62fd0993edaf3"
server_key_iv: "af413c7922b1703dcd90f4504cb22158"
server_key_version: "encryptionkey@0"
I20260929 18:01:57.938211 25012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance:
uuid: "bdc774ac71a847c4853bd7b3cbbd620b"
format_stamp: "Formatted at 2026-09-29 18:01:57 on dist-test-slave-r7ph"
server_key: "e5f5491f9bbfe3ff00a62fd0993edaf3"
server_key_iv: "af413c7922b1703dcd90f4504cb22158"
server_key_version: "encryptionkey@0"
I20260929 18:01:57.942513 25012 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 18:01:57.945624 25027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:57.947196 25012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:01:57.947342 25012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "bdc774ac71a847c4853bd7b3cbbd620b"
format_stamp: "Formatted at 2026-09-29 18:01:57 on dist-test-slave-r7ph"
server_key: "e5f5491f9bbfe3ff00a62fd0993edaf3"
server_key_iv: "af413c7922b1703dcd90f4504cb22158"
server_key_version: "encryptionkey@0"
I20260929 18:01:57.947485 25012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:01:57.960460 25012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:01:57.964445 25012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:01:57.964713 25012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:01:57.965425 25012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:01:57.966499 25012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:01:57.966563 25012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:57.966609 25012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:01:57.966661 25012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:57.977042 25012 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:46671
I20260929 18:01:57.977129 25140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:46671 every 8 connection(s)
I20260929 18:01:57.978312 25012 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 18:01:57.987736 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 25012
I20260929 18:01:57.987901 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance
I20260929 18:01:57.988224 7640 external_mini_cluster.cc:1428] Setting key cfdf6335b195c9d52a8c05fab314f0d9
Sep 29 18:01:57 dist-test-slave-r7ph krb5kdc[24637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704917, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:01:57.994781 24721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:57.980524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:41747 (local address 127.7.118.62:40583)
0929 18:01:57.980666 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:57.980670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:57.981577 (+ 907us) server_negotiation.cc:407] Connection header received
0929 18:01:57.983101 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:57.983104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:57.983178 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:57.983261 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:01:57.984995 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:57.985548 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:57.986916 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:57.987171 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:57.987296 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:57.990642 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:01:57.990646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:01:57.990649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:01:57.990686 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:01:57.992323 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:01:57.992995 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:01:57.992998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:01:57.992999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:01:57.993084 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:01:57.993488 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:01:57.993491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:01:57.993493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:01:57.993703 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:01:57.993817 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:57.994467 (+ 650us) server_negotiation.cc:299] Negotiation successful
0929 18:01:57.994612 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:01:57.995875 25143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:57.980884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:40583 (local address 127.7.118.3:41747)
0929 18:01:57.981397 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:57.981454 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:57.982822 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:57.983460 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:01:57.983469 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:01:57.984153 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:01:57.984776 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:57.984793 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:57.985705 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:57.985716 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:57.986756 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:57.986769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:57.987350 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:57.987354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:57.987614 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:57.988467 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:01:57.988491 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:01:57.990449 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:01:57.992528 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:01:57.992535 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:01:57.992552 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:01:57.992853 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:01:57.993186 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:01:57.993190 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:01:57.993192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:01:57.993374 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:01:57.993833 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:01:57.993842 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:01:57.994146 (+ 304us) client_negotiation.cc:769] Sending connection context
0929 18:01:57.994408 (+ 262us) client_negotiation.cc:240] Negotiation successful
0929 18:01:57.994829 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":126,"threads_started":1}
I20260929 18:01:57.997311 25141 heartbeater.cc:347] Connected to a master server at 127.7.118.62:40583
I20260929 18:01:57.997574 25141 heartbeater.cc:464] Registering TS with master...
I20260929 18:01:57.998091 25141 heartbeater.cc:511] Master 127.7.118.62:40583 requested a full tablet report, sending...
I20260929 18:01:57.999594 24682 ts_manager.cc:194] Registered new tserver with Master: bdc774ac71a847c4853bd7b3cbbd620b (127.7.118.3:46671)
I20260929 18:01:58.000202 24682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.3@KRBTEST.COM'} at 127.7.118.3:41747
I20260929 18:01:58.003541 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:01:58.014072 24721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:58.005850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47044 (local address 127.7.118.62:40583)
0929 18:01:58.005998 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:58.006002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:58.006151 (+ 149us) server_negotiation.cc:407] Connection header received
0929 18:01:58.006317 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:58.006320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:58.006364 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:58.006437 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:01:58.007190 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.007765 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.008714 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.009088 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.009184 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:58.010757 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:01:58.010760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:01:58.010762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:01:58.010790 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:01:58.012391 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:01:58.012816 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:01:58.012819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:01:58.012820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:01:58.012898 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:01:58.013209 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:01:58.013211 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:01:58.013212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:01:58.013434 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:01:58.013536 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:58.013802 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 18:01:58.013930 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 18:01:58.017309 24682 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:01:58.020486 24682 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:01:58.036520 25155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:58.030221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:46671 (local address 127.0.0.1:59994)
0929 18:01:58.031021 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:58.031033 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:58.031107 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:58.031926 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:01:58.031931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:01:58.031952 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:01:58.032059 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:58.032066 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.033362 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.033366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:58.034610 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:58.034619 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.035745 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.035748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:58.035936 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:58.036057 (+ 121us) client_negotiation.cc:769] Sending connection context
0929 18:01:58.036176 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 18:01:58.036346 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":93,"thread_start_us":160,"threads_started":1}
I20260929 18:01:58.036528 25156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:58.031056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59994 (local address 127.7.118.3:46671)
0929 18:01:58.031458 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:58.031469 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:58.031492 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:01:58.031564 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:58.031569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:58.031768 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:58.031928 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:01:58.032159 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.033229 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.034744 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.035628 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.035756 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:58.035908 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:58.036238 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 18:01:58.036346 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":131,"threads_started":1}
I20260929 18:01:58.036566 25154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:58.030557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38290 (local address 127.7.118.1:33795)
0929 18:01:58.030861 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:58.030868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:58.030892 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:01:58.030974 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:58.030980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:58.031213 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:58.031380 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:01:58.031778 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.033074 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.034438 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.035342 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.035453 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:58.035640 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:58.036326 (+ 686us) server_negotiation.cc:299] Negotiation successful
0929 18:01:58.036428 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":73,"threads_started":1}
I20260929 18:01:58.036922 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:58.029895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:33795 (local address 127.0.0.1:38290)
0929 18:01:58.030435 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:58.030493 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:58.030625 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:58.031403 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:01:58.031409 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:01:58.031435 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:01:58.031630 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:58.031638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.033207 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.033211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:58.034229 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:58.034240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.035723 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.035728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:58.035938 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:58.036060 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 18:01:58.036173 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 18:01:58.036792 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":561,"thread_start_us":95,"threads_started":1}
I20260929 18:01:58.037669 25153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:58.030731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:46193 (local address 127.0.0.1:57836)
0929 18:01:58.032396 (+ 1665us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:58.032410 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:58.032504 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:58.032947 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:01:58.032950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:01:58.032967 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:01:58.033069 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:58.033075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.034511 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.034515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:58.035697 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:58.035709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.037280 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.037283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:58.037370 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:58.037393 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:01:58.037472 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 18:01:58.037539 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1605,"thread_start_us":87,"threads_started":1}
I20260929 18:01:58.037791 25157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:58.031452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57836 (local address 127.7.118.2:46193)
0929 18:01:58.031760 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:58.031771 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:58.032604 (+ 833us) server_negotiation.cc:407] Connection header received
0929 18:01:58.032672 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:58.032676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:58.032813 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:58.032976 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:01:58.033166 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.034363 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.035833 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.036616 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.037069 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:58.037217 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:58.037532 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 18:01:58.037636 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":130,"threads_started":1}
I20260929 18:01:58.039875 24807 tablet_service.cc:1467] Processing CreateTablet for tablet 47dfeb56f080458da9c57f7df9a7f939 (DEFAULT_TABLE table=test-table [id=a403ff84c6c04ded824683bd7e97fa4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:01:58.039969 25075 tablet_service.cc:1467] Processing CreateTablet for tablet 47dfeb56f080458da9c57f7df9a7f939 (DEFAULT_TABLE table=test-table [id=a403ff84c6c04ded824683bd7e97fa4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:01:58.040910 24807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47dfeb56f080458da9c57f7df9a7f939. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:01:58.040599 24941 tablet_service.cc:1467] Processing CreateTablet for tablet 47dfeb56f080458da9c57f7df9a7f939 (DEFAULT_TABLE table=test-table [id=a403ff84c6c04ded824683bd7e97fa4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:01:58.041025 25075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47dfeb56f080458da9c57f7df9a7f939. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:01:58.041540 24941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47dfeb56f080458da9c57f7df9a7f939. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:01:58.048750 25160 tablet_bootstrap.cc:492] T 47dfeb56f080458da9c57f7df9a7f939 P 4526b6aced19411d8fc19800b5dfb8f3: Bootstrap starting.
I20260929 18:01:58.050668 25158 tablet_bootstrap.cc:492] T 47dfeb56f080458da9c57f7df9a7f939 P c61f5856655948a79b1d60f6db3e2ee3: Bootstrap starting.
I20260929 18:01:58.050725 25159 tablet_bootstrap.cc:492] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b: Bootstrap starting.
I20260929 18:01:58.051209 25160 tablet_bootstrap.cc:654] T 47dfeb56f080458da9c57f7df9a7f939 P 4526b6aced19411d8fc19800b5dfb8f3: Neither blocks nor log segments found. Creating new log.
I20260929 18:01:58.052588 25160 log.cc:824] T 47dfeb56f080458da9c57f7df9a7f939 P 4526b6aced19411d8fc19800b5dfb8f3: Log is configured to *not* fsync() on all Append() calls
I20260929 18:01:58.053112 25159 tablet_bootstrap.cc:654] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b: Neither blocks nor log segments found. Creating new log.
I20260929 18:01:58.053113 25158 tablet_bootstrap.cc:654] T 47dfeb56f080458da9c57f7df9a7f939 P c61f5856655948a79b1d60f6db3e2ee3: Neither blocks nor log segments found. Creating new log.
I20260929 18:01:58.053983 25158 log.cc:824] T 47dfeb56f080458da9c57f7df9a7f939 P c61f5856655948a79b1d60f6db3e2ee3: Log is configured to *not* fsync() on all Append() calls
I20260929 18:01:58.054518 25159 log.cc:824] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b: Log is configured to *not* fsync() on all Append() calls
I20260929 18:01:58.055058 25160 tablet_bootstrap.cc:492] T 47dfeb56f080458da9c57f7df9a7f939 P 4526b6aced19411d8fc19800b5dfb8f3: No bootstrap required, opened a new log
I20260929 18:01:58.055263 25160 ts_tablet_manager.cc:1397] T 47dfeb56f080458da9c57f7df9a7f939 P 4526b6aced19411d8fc19800b5dfb8f3: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260929 18:01:58.056079 25158 tablet_bootstrap.cc:492] T 47dfeb56f080458da9c57f7df9a7f939 P c61f5856655948a79b1d60f6db3e2ee3: No bootstrap required, opened a new log
I20260929 18:01:58.056263 25158 ts_tablet_manager.cc:1397] T 47dfeb56f080458da9c57f7df9a7f939 P c61f5856655948a79b1d60f6db3e2ee3: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 18:01:58.056269 25159 tablet_bootstrap.cc:492] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b: No bootstrap required, opened a new log
I20260929 18:01:58.056407 25159 ts_tablet_manager.cc:1397] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:01:58.059485 25159 raft_consensus.cc:348] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61f5856655948a79b1d60f6db3e2ee3" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33795 } } peers { permanent_uuid: "bdc774ac71a847c4853bd7b3cbbd620b" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 46671 } } peers { permanent_uuid: "4526b6aced19411d8fc19800b5dfb8f3" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 46193 } }
I20260929 18:01:58.059518 25160 raft_consensus.cc:348] T 47dfeb56f080458da9c57f7df9a7f939 P 4526b6aced19411d8fc19800b5dfb8f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61f5856655948a79b1d60f6db3e2ee3" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33795 } } peers { permanent_uuid: "bdc774ac71a847c4853bd7b3cbbd620b" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 46671 } } peers { permanent_uuid: "4526b6aced19411d8fc19800b5dfb8f3" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 46193 } }
I20260929 18:01:58.059773 25159 raft_consensus.cc:374] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:01:58.059810 25160 raft_consensus.cc:374] T 47dfeb56f080458da9c57f7df9a7f939 P 4526b6aced19411d8fc19800b5dfb8f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:01:58.059847 25159 raft_consensus.cc:729] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdc774ac71a847c4853bd7b3cbbd620b, State: Initialized, Role: FOLLOWER
I20260929 18:01:58.059917 25160 raft_consensus.cc:729] T 47dfeb56f080458da9c57f7df9a7f939 P 4526b6aced19411d8fc19800b5dfb8f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4526b6aced19411d8fc19800b5dfb8f3, State: Initialized, Role: FOLLOWER
I20260929 18:01:58.060392 25159 consensus_queue.cc:260] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61f5856655948a79b1d60f6db3e2ee3" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33795 } } peers { permanent_uuid: "bdc774ac71a847c4853bd7b3cbbd620b" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 46671 } } peers { permanent_uuid: "4526b6aced19411d8fc19800b5dfb8f3" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 46193 } }
I20260929 18:01:58.060392 25160 consensus_queue.cc:260] T 47dfeb56f080458da9c57f7df9a7f939 P 4526b6aced19411d8fc19800b5dfb8f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61f5856655948a79b1d60f6db3e2ee3" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33795 } } peers { permanent_uuid: "bdc774ac71a847c4853bd7b3cbbd620b" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 46671 } } peers { permanent_uuid: "4526b6aced19411d8fc19800b5dfb8f3" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 46193 } }
I20260929 18:01:58.061111 25158 raft_consensus.cc:348] T 47dfeb56f080458da9c57f7df9a7f939 P c61f5856655948a79b1d60f6db3e2ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61f5856655948a79b1d60f6db3e2ee3" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33795 } } peers { permanent_uuid: "bdc774ac71a847c4853bd7b3cbbd620b" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 46671 } } peers { permanent_uuid: "4526b6aced19411d8fc19800b5dfb8f3" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 46193 } }
I20260929 18:01:58.061403 25158 raft_consensus.cc:374] T 47dfeb56f080458da9c57f7df9a7f939 P c61f5856655948a79b1d60f6db3e2ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:01:58.061499 25158 raft_consensus.cc:729] T 47dfeb56f080458da9c57f7df9a7f939 P c61f5856655948a79b1d60f6db3e2ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c61f5856655948a79b1d60f6db3e2ee3, State: Initialized, Role: FOLLOWER
I20260929 18:01:58.061600 25160 ts_tablet_manager.cc:1428] T 47dfeb56f080458da9c57f7df9a7f939 P 4526b6aced19411d8fc19800b5dfb8f3: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260929 18:01:58.061594 25159 ts_tablet_manager.cc:1428] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:01:58.061975 25141 heartbeater.cc:503] Master 127.7.118.62:40583 was elected leader, sending a full tablet report...
I20260929 18:01:58.061960 25158 consensus_queue.cc:260] T 47dfeb56f080458da9c57f7df9a7f939 P c61f5856655948a79b1d60f6db3e2ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61f5856655948a79b1d60f6db3e2ee3" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33795 } } peers { permanent_uuid: "bdc774ac71a847c4853bd7b3cbbd620b" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 46671 } } peers { permanent_uuid: "4526b6aced19411d8fc19800b5dfb8f3" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 46193 } }
I20260929 18:01:58.062054 25007 heartbeater.cc:503] Master 127.7.118.62:40583 was elected leader, sending a full tablet report...
I20260929 18:01:58.062693 24873 heartbeater.cc:503] Master 127.7.118.62:40583 was elected leader, sending a full tablet report...
I20260929 18:01:58.063035 25158 ts_tablet_manager.cc:1428] T 47dfeb56f080458da9c57f7df9a7f939 P c61f5856655948a79b1d60f6db3e2ee3: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
W20260929 18:01:58.229630 25142 tablet.cc:2369] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:01:58.242023 25164 raft_consensus.cc:482] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:01:58.242316 25164 raft_consensus.cc:504] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61f5856655948a79b1d60f6db3e2ee3" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33795 } } peers { permanent_uuid: "bdc774ac71a847c4853bd7b3cbbd620b" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 46671 } } peers { permanent_uuid: "4526b6aced19411d8fc19800b5dfb8f3" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 46193 } }
I20260929 18:01:58.243602 25164 leader_election.cc:288] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c61f5856655948a79b1d60f6db3e2ee3 (127.7.118.1:33795), 4526b6aced19411d8fc19800b5dfb8f3 (127.7.118.2:46193)
W20260929 18:01:58.245791 25008 tablet.cc:2369] T 47dfeb56f080458da9c57f7df9a7f939 P 4526b6aced19411d8fc19800b5dfb8f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:01:58.248162 25154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:58.244076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:42189 (local address 127.7.118.1:33795)
0929 18:01:58.244234 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:58.244241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:58.244261 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:01:58.244333 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:58.244336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:58.244384 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:58.244558 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:01:58.244891 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.245568 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.246905 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.247508 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.247623 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:58.247695 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:58.247928 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 18:01:58.248009 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:01:58.248165 25143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:58.243952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:33795 (local address 127.7.118.3:42189)
0929 18:01:58.244104 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:58.244122 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:58.244227 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:58.244609 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:01:58.244617 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:01:58.244636 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:01:58.244737 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:58.244742 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.245676 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.245682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:58.246773 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:58.246783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.247662 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.247665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:58.247755 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:58.247776 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:01:58.247896 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 18:01:58.247981 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":20}
I20260929 18:01:58.248278 25157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:58.244413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:56821 (local address 127.7.118.2:46193)
0929 18:01:58.244547 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:58.244553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:58.244792 (+ 239us) server_negotiation.cc:407] Connection header received
0929 18:01:58.244901 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:58.244903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:58.244942 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:58.244999 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:01:58.245276 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.245894 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.247152 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.247741 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.247822 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:58.247894 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:58.248086 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 18:01:58.248158 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:01:58.248323 25167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:58.244258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:46193 (local address 127.7.118.3:56821)
0929 18:01:58.244696 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:58.244708 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:58.244792 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:58.245028 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:01:58.245030 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:01:58.245047 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:01:58.245161 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:58.245167 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.246060 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.246063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:58.247002 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:58.247038 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.247862 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.247868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:58.247971 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:58.247989 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 18:01:58.248061 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 18:01:58.248119 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":1,"thread_start_us":125,"threads_started":1}
I20260929 18:01:58.248936 24815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47dfeb56f080458da9c57f7df9a7f939" candidate_uuid: "bdc774ac71a847c4853bd7b3cbbd620b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c61f5856655948a79b1d60f6db3e2ee3" is_pre_election: true
I20260929 18:01:58.248943 24961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47dfeb56f080458da9c57f7df9a7f939" candidate_uuid: "bdc774ac71a847c4853bd7b3cbbd620b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4526b6aced19411d8fc19800b5dfb8f3" is_pre_election: true
I20260929 18:01:58.249261 24815 raft_consensus.cc:2454] T 47dfeb56f080458da9c57f7df9a7f939 P c61f5856655948a79b1d60f6db3e2ee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdc774ac71a847c4853bd7b3cbbd620b in term 0.
I20260929 18:01:58.249264 24961 raft_consensus.cc:2454] T 47dfeb56f080458da9c57f7df9a7f939 P 4526b6aced19411d8fc19800b5dfb8f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdc774ac71a847c4853bd7b3cbbd620b in term 0.
I20260929 18:01:58.249784 25030 leader_election.cc:302] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4526b6aced19411d8fc19800b5dfb8f3, bdc774ac71a847c4853bd7b3cbbd620b; no voters:
I20260929 18:01:58.250079 25164 raft_consensus.cc:2781] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:01:58.250160 25164 raft_consensus.cc:482] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:01:58.250191 25164 raft_consensus.cc:3037] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:01:58.251380 25164 raft_consensus.cc:504] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61f5856655948a79b1d60f6db3e2ee3" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33795 } } peers { permanent_uuid: "bdc774ac71a847c4853bd7b3cbbd620b" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 46671 } } peers { permanent_uuid: "4526b6aced19411d8fc19800b5dfb8f3" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 46193 } }
I20260929 18:01:58.251753 25164 leader_election.cc:288] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b [CANDIDATE]: Term 1 election: Requested vote from peers c61f5856655948a79b1d60f6db3e2ee3 (127.7.118.1:33795), 4526b6aced19411d8fc19800b5dfb8f3 (127.7.118.2:46193)
I20260929 18:01:58.252279 24815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47dfeb56f080458da9c57f7df9a7f939" candidate_uuid: "bdc774ac71a847c4853bd7b3cbbd620b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c61f5856655948a79b1d60f6db3e2ee3"
I20260929 18:01:58.252393 24961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47dfeb56f080458da9c57f7df9a7f939" candidate_uuid: "bdc774ac71a847c4853bd7b3cbbd620b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4526b6aced19411d8fc19800b5dfb8f3"
I20260929 18:01:58.252457 24815 raft_consensus.cc:3037] T 47dfeb56f080458da9c57f7df9a7f939 P c61f5856655948a79b1d60f6db3e2ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:01:58.252614 24961 raft_consensus.cc:3037] T 47dfeb56f080458da9c57f7df9a7f939 P 4526b6aced19411d8fc19800b5dfb8f3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:01:58.253683 24815 raft_consensus.cc:2454] T 47dfeb56f080458da9c57f7df9a7f939 P c61f5856655948a79b1d60f6db3e2ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdc774ac71a847c4853bd7b3cbbd620b in term 1.
I20260929 18:01:58.253824 24961 raft_consensus.cc:2454] T 47dfeb56f080458da9c57f7df9a7f939 P 4526b6aced19411d8fc19800b5dfb8f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdc774ac71a847c4853bd7b3cbbd620b in term 1.
I20260929 18:01:58.254086 25030 leader_election.cc:302] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdc774ac71a847c4853bd7b3cbbd620b, c61f5856655948a79b1d60f6db3e2ee3; no voters:
I20260929 18:01:58.254330 25164 raft_consensus.cc:2781] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:01:58.254590 25164 raft_consensus.cc:686] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b [term 1 LEADER]: Becoming Leader. State: Replica: bdc774ac71a847c4853bd7b3cbbd620b, State: Running, Role: LEADER
I20260929 18:01:58.254937 25164 consensus_queue.cc:237] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61f5856655948a79b1d60f6db3e2ee3" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33795 } } peers { permanent_uuid: "bdc774ac71a847c4853bd7b3cbbd620b" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 46671 } } peers { permanent_uuid: "4526b6aced19411d8fc19800b5dfb8f3" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 46193 } }
W20260929 18:01:58.255046 24874 tablet.cc:2369] T 47dfeb56f080458da9c57f7df9a7f939 P c61f5856655948a79b1d60f6db3e2ee3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:01:58.258107 24682 catalog_manager.cc:5495] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b reported cstate change: term changed from 0 to 1, leader changed from <none> to bdc774ac71a847c4853bd7b3cbbd620b (127.7.118.3). New cstate: current_term: 1 leader_uuid: "bdc774ac71a847c4853bd7b3cbbd620b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61f5856655948a79b1d60f6db3e2ee3" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdc774ac71a847c4853bd7b3cbbd620b" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 46671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4526b6aced19411d8fc19800b5dfb8f3" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 46193 } health_report { overall_health: UNKNOWN } } }
I20260929 18:01:58.276613 25156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:58.272293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60008 (local address 127.7.118.3:46671)
0929 18:01:58.272425 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:58.272428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:58.272444 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:01:58.272523 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:58.272525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:58.272606 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:58.272697 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:01:58.272942 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.273782 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.274789 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.275019 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.275113 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:58.275503 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:01:58.275964 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:01:58.276070 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:58.276210 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 18:01:58.276325 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:01:58.283601 24815 raft_consensus.cc:1261] T 47dfeb56f080458da9c57f7df9a7f939 P c61f5856655948a79b1d60f6db3e2ee3 [term 1 FOLLOWER]: Refusing update from remote peer bdc774ac71a847c4853bd7b3cbbd620b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:01:58.283602 24961 raft_consensus.cc:1261] T 47dfeb56f080458da9c57f7df9a7f939 P 4526b6aced19411d8fc19800b5dfb8f3 [term 1 FOLLOWER]: Refusing update from remote peer bdc774ac71a847c4853bd7b3cbbd620b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:01:58.284451 25164 consensus_queue.cc:1035] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4526b6aced19411d8fc19800b5dfb8f3" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:01:58.284732 25168 consensus_queue.cc:1035] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c61f5856655948a79b1d60f6db3e2ee3" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 33795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:01:58.302510 24681 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:01:58.302735 24681 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:01:58.305922 24681 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b47e515197af444891a7f0e979048bb7: Sending DeleteTablet for 3 replicas of tablet 47dfeb56f080458da9c57f7df9a7f939
I20260929 18:01:58.306910 25075 tablet_service.cc:1514] Processing DeleteTablet for tablet 47dfeb56f080458da9c57f7df9a7f939 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:01:58 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:59994
I20260929 18:01:58.307170 24807 tablet_service.cc:1514] Processing DeleteTablet for tablet 47dfeb56f080458da9c57f7df9a7f939 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:01:58 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:38290
I20260929 18:01:58.307492 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 24744
I20260929 18:01:58.307696 25180 tablet_replica.cc:330] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b: stopping tablet replica
I20260929 18:01:58.307703 24941 tablet_service.cc:1514] Processing DeleteTablet for tablet 47dfeb56f080458da9c57f7df9a7f939 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:01:58 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:57836
I20260929 18:01:58.308110 25180 raft_consensus.cc:2229] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:01:58.308534 25180 raft_consensus.cc:2258] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:01:58.310345 25180 ts_tablet_manager.cc:1905] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:01:58.310515 25182 tablet_replica.cc:330] T 47dfeb56f080458da9c57f7df9a7f939 P 4526b6aced19411d8fc19800b5dfb8f3: stopping tablet replica
I20260929 18:01:58.310905 25182 raft_consensus.cc:2229] T 47dfeb56f080458da9c57f7df9a7f939 P 4526b6aced19411d8fc19800b5dfb8f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:01:58.311327 25182 raft_consensus.cc:2258] T 47dfeb56f080458da9c57f7df9a7f939 P 4526b6aced19411d8fc19800b5dfb8f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:01:58.313951 25180 ts_tablet_manager.cc:1918] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:01:58.314080 25180 log.cc:1196] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704657566865-7640-0/minicluster-data/ts-2/wal/wals/47dfeb56f080458da9c57f7df9a7f939
I20260929 18:01:58.314519 25180 ts_tablet_manager.cc:1939] T 47dfeb56f080458da9c57f7df9a7f939 P bdc774ac71a847c4853bd7b3cbbd620b: Deleting consensus metadata
I20260929 18:01:58.316347 24669 catalog_manager.cc:4841] TS bdc774ac71a847c4853bd7b3cbbd620b (127.7.118.3:46671): tablet 47dfeb56f080458da9c57f7df9a7f939 (table test-table [id=a403ff84c6c04ded824683bd7e97fa4f]) successfully deleted
I20260929 18:01:58.316418 25182 ts_tablet_manager.cc:1905] T 47dfeb56f080458da9c57f7df9a7f939 P 4526b6aced19411d8fc19800b5dfb8f3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 18:01:58.317170 24668 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.118.1:33795 (error 108)
I20260929 18:01:58.318300 25153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:58.317921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:33795 (local address 127.0.0.1:38298)
0929 18:01:58.318065 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:58.318143 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.1:33795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 18:01:58.318383 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 24878
W20260929 18:01:58.320602 24668 catalog_manager.cc:4584] TS c61f5856655948a79b1d60f6db3e2ee3 (127.7.118.1:33795): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 47dfeb56f080458da9c57f7df9a7f939: Network error: Client connection negotiation failed: client connection to 127.7.118.1:33795: connect: Connection refused (error 111)
I20260929 18:01:58.325160 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 25012
I20260929 18:01:58.327196 25153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:58.326793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:46193 (local address 127.0.0.1:57852)
0929 18:01:58.326973 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:58.327046 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.2:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 18:01:58.331636 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 24650
2026-09-29T18:01:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2527 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T18:01:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:01:58Z Disabled control of system clock
I20260929 18:01:58.363821 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:35567
--webserver_interface=127.7.118.62
--webserver_port=0
--builtin_ntp_servers=127.7.118.20:36753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:35567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:01:58.485172 25187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:01:58.485499 25187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:01:58.485538 25187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:01:58.490029 25187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:01:58.490108 25187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:01:58.490123 25187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:01:58.490142 25187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:01:58.490173 25187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:01:58.495716 25187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:35567
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:35567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:01:58.496927 25187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:01:58.498517 25187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:01:58.508105 25195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:01:58.508240 25187 server_base.cc:1034] running on GCE node
W20260929 18:01:58.508108 25192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:01:58.508460 25193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:01:58.509162 25187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:01:58.510519 25187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:01:58.511705 25187 hybrid_clock.cc:648] HybridClock initialized: now 1790704918511656 us; error 74 us; skew 500 ppm
I20260929 18:01:58.513907 25187 webserver.cc:460] Webserver started at http://127.7.118.62:37929/ using document root <none> and password file <none>
I20260929 18:01:58.514668 25187 fs_manager.cc:359] Metadata directory not provided
I20260929 18:01:58.514742 25187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:01:58.514985 25187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:01:58.517252 25187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "f6dc48850eec486fa7aac1ea84653b6f"
format_stamp: "Formatted at 2026-09-29 18:01:58 on dist-test-slave-r7ph"
server_key: "aecc475461be488aa1660e1b7b991b6f"
server_key_iv: "16f2d7077b509770db05ecf2e377e225"
server_key_version: "encryptionkey@0"
I20260929 18:01:58.517874 25187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "f6dc48850eec486fa7aac1ea84653b6f"
format_stamp: "Formatted at 2026-09-29 18:01:58 on dist-test-slave-r7ph"
server_key: "aecc475461be488aa1660e1b7b991b6f"
server_key_iv: "16f2d7077b509770db05ecf2e377e225"
server_key_version: "encryptionkey@0"
I20260929 18:01:58.522018 25187 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:01:58.524937 25201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:58.526583 25187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:01:58.526764 25187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "f6dc48850eec486fa7aac1ea84653b6f"
format_stamp: "Formatted at 2026-09-29 18:01:58 on dist-test-slave-r7ph"
server_key: "aecc475461be488aa1660e1b7b991b6f"
server_key_iv: "16f2d7077b509770db05ecf2e377e225"
server_key_version: "encryptionkey@0"
I20260929 18:01:58.526993 25187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:01:58.551918 25187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:01:58.552657 25187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:01:58.552821 25187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:01:58.560606 25187 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:35567
I20260929 18:01:58.560614 25253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:35567 every 8 connection(s)
I20260929 18:01:58.561714 25187 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:01:58.565330 25254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:01:58.571326 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 25187
I20260929 18:01:58.571545 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:01:58.571797 25254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6dc48850eec486fa7aac1ea84653b6f: Bootstrap starting.
I20260929 18:01:58.572077 7640 external_mini_cluster.cc:1428] Setting key 84e66d7e4b9462a08b4c243151b33145
I20260929 18:01:58.575060 25254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6dc48850eec486fa7aac1ea84653b6f: Neither blocks nor log segments found. Creating new log.
I20260929 18:01:58.575943 25254 log.cc:824] T 00000000000000000000000000000000 P f6dc48850eec486fa7aac1ea84653b6f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:01:58.578493 25254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6dc48850eec486fa7aac1ea84653b6f: No bootstrap required, opened a new log
I20260929 18:01:58.583333 25254 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6dc48850eec486fa7aac1ea84653b6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6dc48850eec486fa7aac1ea84653b6f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35567 } }
I20260929 18:01:58.583566 25257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:58.573827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35734 (local address 127.7.118.62:35567)
0929 18:01:58.574615 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:58.574631 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:58.574678 (+ 47us) server_negotiation.cc:407] Connection header received
0929 18:01:58.575528 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:58.575551 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:58.576063 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:58.576480 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:01:58.576992 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.578058 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.579417 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.580029 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.580182 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:58.581185 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:01:58.581191 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:01:58.581207 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:01:58.581724 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:01:58.581789 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:01:58.581814 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:01:58.581900 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:58.582161 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 18:01:58.582622 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":158,"threads_started":1}
I20260929 18:01:58.583637 25254 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6dc48850eec486fa7aac1ea84653b6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:01:58.583679 25254 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6dc48850eec486fa7aac1ea84653b6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6dc48850eec486fa7aac1ea84653b6f, State: Initialized, Role: FOLLOWER
I20260929 18:01:58.584249 25254 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6dc48850eec486fa7aac1ea84653b6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6dc48850eec486fa7aac1ea84653b6f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35567 } }
I20260929 18:01:58.584455 25254 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6dc48850eec486fa7aac1ea84653b6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:01:58.584558 25254 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6dc48850eec486fa7aac1ea84653b6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:01:58.584712 25254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6dc48850eec486fa7aac1ea84653b6f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:01:58.586292 25254 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6dc48850eec486fa7aac1ea84653b6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6dc48850eec486fa7aac1ea84653b6f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35567 } }
I20260929 18:01:58.586753 25254 leader_election.cc:302] T 00000000000000000000000000000000 P f6dc48850eec486fa7aac1ea84653b6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6dc48850eec486fa7aac1ea84653b6f; no voters:
I20260929 18:01:58.587195 25254 leader_election.cc:288] T 00000000000000000000000000000000 P f6dc48850eec486fa7aac1ea84653b6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:01:58.587633 25259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6dc48850eec486fa7aac1ea84653b6f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:01:58.588038 25259 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6dc48850eec486fa7aac1ea84653b6f [term 1 LEADER]: Becoming Leader. State: Replica: f6dc48850eec486fa7aac1ea84653b6f, State: Running, Role: LEADER
I20260929 18:01:58.588395 25259 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6dc48850eec486fa7aac1ea84653b6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6dc48850eec486fa7aac1ea84653b6f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35567 } }
I20260929 18:01:58.588820 25254 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6dc48850eec486fa7aac1ea84653b6f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:01:58.590423 25260 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6dc48850eec486fa7aac1ea84653b6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6dc48850eec486fa7aac1ea84653b6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6dc48850eec486fa7aac1ea84653b6f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35567 } } }
I20260929 18:01:58.590590 25260 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6dc48850eec486fa7aac1ea84653b6f [sys.catalog]: This master's current role is: LEADER
I20260929 18:01:58.591145 25269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:01:58.591537 25261 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6dc48850eec486fa7aac1ea84653b6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6dc48850eec486fa7aac1ea84653b6f. Latest consensus state: current_term: 1 leader_uuid: "f6dc48850eec486fa7aac1ea84653b6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6dc48850eec486fa7aac1ea84653b6f" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35567 } } }
I20260929 18:01:58.591660 25261 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6dc48850eec486fa7aac1ea84653b6f [sys.catalog]: This master's current role is: LEADER
I20260929 18:01:58.594620 25269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:01:58.601500 25269 catalog_manager.cc:1347] Generated new cluster ID: f338ca50730c4f73954711f42bcc4644
I20260929 18:01:58.601636 25269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:01:58.619819 25269 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:01:58.621227 25269 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:01:58.633085 25269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6dc48850eec486fa7aac1ea84653b6f: Generated new TSK 0
I20260929 18:01:58.634071 25269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:01:58.643338 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:0
--local_ip_for_outbound_sockets=127.7.118.1
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:35567
--builtin_ntp_servers=127.7.118.20:36753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:01:58.771023 25278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:01:58.771402 25278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:01:58.771488 25278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:01:58.775832 25278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:01:58.775951 25278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:01:58.776057 25278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 18:01:58.781337 25278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:35567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:01:58.782660 25278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:01:58.784209 25278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:01:58.795902 25284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:01:58.796125 25278 server_base.cc:1034] running on GCE node
W20260929 18:01:58.796022 25283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:01:58.796010 25286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:01:58.796813 25278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:01:58.797648 25278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:01:58.798861 25278 hybrid_clock.cc:648] HybridClock initialized: now 1790704918798843 us; error 34 us; skew 500 ppm
I20260929 18:01:58.801420 25278 webserver.cc:460] Webserver started at http://127.7.118.1:41779/ using document root <none> and password file <none>
I20260929 18:01:58.802146 25278 fs_manager.cc:359] Metadata directory not provided
I20260929 18:01:58.802232 25278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:01:58.802552 25278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:01:58.805012 25278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-0/data/instance:
uuid: "63c04688e9d34e0386d15bcbf92316d5"
format_stamp: "Formatted at 2026-09-29 18:01:58 on dist-test-slave-r7ph"
server_key: "3d15689389c7664f1beecf3230a4c190"
server_key_iv: "e834697884c3b15cbde181c8991fba03"
server_key_version: "encryptionkey@0"
I20260929 18:01:58.805727 25278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance:
uuid: "63c04688e9d34e0386d15bcbf92316d5"
format_stamp: "Formatted at 2026-09-29 18:01:58 on dist-test-slave-r7ph"
server_key: "3d15689389c7664f1beecf3230a4c190"
server_key_iv: "e834697884c3b15cbde181c8991fba03"
server_key_version: "encryptionkey@0"
I20260929 18:01:58.810956 25278 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 18:01:58.814327 25292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:58.816150 25278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:01:58.816381 25278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "63c04688e9d34e0386d15bcbf92316d5"
format_stamp: "Formatted at 2026-09-29 18:01:58 on dist-test-slave-r7ph"
server_key: "3d15689389c7664f1beecf3230a4c190"
server_key_iv: "e834697884c3b15cbde181c8991fba03"
server_key_version: "encryptionkey@0"
I20260929 18:01:58.816619 25278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:01:58.834843 25278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:01:58.835681 25278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:01:58.835914 25278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:01:58.836599 25278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:01:58.837687 25278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:01:58.837765 25278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:58.837845 25278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:01:58.837893 25278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:58.848567 25278 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:35665
I20260929 18:01:58.848594 25405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:35665 every 8 connection(s)
I20260929 18:01:58.849656 25278 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 18:01:58.852186 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 25278
I20260929 18:01:58.852313 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance
I20260929 18:01:58.852635 7640 external_mini_cluster.cc:1428] Setting key 173f42b9a3ed4c6531c4e5181a8eebba
I20260929 18:01:58.854897 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:0
--local_ip_for_outbound_sockets=127.7.118.2
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:35567
--builtin_ntp_servers=127.7.118.20:36753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:01:58.860977 25257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:58.852060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:44667 (local address 127.7.118.62:35567)
0929 18:01:58.852211 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:58.852215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:58.853750 (+ 1535us) server_negotiation.cc:407] Connection header received
0929 18:01:58.855588 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:58.855592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:58.855649 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:58.855730 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:01:58.856726 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.857271 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.858640 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.858881 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.859005 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:58.860347 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:01:58.860351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:01:58.860353 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:01:58.860367 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:01:58.860387 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:01:58.860395 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:01:58.860528 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:58.860739 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 18:01:58.860818 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:01:58.862080 25408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:58.852633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:35567 (local address 127.7.118.1:44667)
0929 18:01:58.853552 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:58.853622 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:58.855320 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:58.855934 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:01:58.855942 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:01:58.856038 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:01:58.856577 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:58.856590 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.857424 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.857427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:58.858500 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:58.858511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:58.859059 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:58.859062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:58.859246 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:58.860023 (+ 777us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:01:58.860025 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:01:58.860078 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:01:58.860156 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:01:58.860528 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:01:58.860566 (+ 38us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:01:58.860576 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 18:01:58.860741 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 18:01:58.861206 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":161,"threads_started":1}
I20260929 18:01:58.863641 25406 heartbeater.cc:347] Connected to a master server at 127.7.118.62:35567
I20260929 18:01:58.863946 25406 heartbeater.cc:464] Registering TS with master...
I20260929 18:01:58.864568 25406 heartbeater.cc:511] Master 127.7.118.62:35567 requested a full tablet report, sending...
I20260929 18:01:58.866395 25218 ts_manager.cc:194] Registered new tserver with Master: 63c04688e9d34e0386d15bcbf92316d5 (127.7.118.1:35665)
I20260929 18:01:58.868027 25218 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.118.1:44667
W20260929 18:01:58.973114 25409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:01:58.973443 25409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:01:58.973484 25409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:01:58.978575 25409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:01:58.978672 25409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:01:58.978739 25409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 18:01:58.984473 25409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:35567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:01:58.985620 25409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:01:58.987097 25409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:01:58.998752 25414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:01:58.998948 25415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:01:58.998888 25409 server_base.cc:1034] running on GCE node
W20260929 18:01:58.999220 25417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:01:58.999770 25409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:01:59.000532 25409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:01:59.001736 25409 hybrid_clock.cc:648] HybridClock initialized: now 1790704919001691 us; error 61 us; skew 500 ppm
I20260929 18:01:59.004462 25409 webserver.cc:460] Webserver started at http://127.7.118.2:42937/ using document root <none> and password file <none>
I20260929 18:01:59.005205 25409 fs_manager.cc:359] Metadata directory not provided
I20260929 18:01:59.005294 25409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:01:59.005553 25409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:01:59.008180 25409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-1/data/instance:
uuid: "f523d91e01b94918a2ed5b43a323bd63"
format_stamp: "Formatted at 2026-09-29 18:01:58 on dist-test-slave-r7ph"
server_key: "cdc76b74067a994735c89472349ddf47"
server_key_iv: "6e6c5160985b5bf19dcb26b32bdc69a1"
server_key_version: "encryptionkey@0"
I20260929 18:01:59.008893 25409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance:
uuid: "f523d91e01b94918a2ed5b43a323bd63"
format_stamp: "Formatted at 2026-09-29 18:01:58 on dist-test-slave-r7ph"
server_key: "cdc76b74067a994735c89472349ddf47"
server_key_iv: "6e6c5160985b5bf19dcb26b32bdc69a1"
server_key_version: "encryptionkey@0"
I20260929 18:01:59.013697 25409 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 18:01:59.016847 25423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:59.018684 25409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:01:59.018878 25409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "f523d91e01b94918a2ed5b43a323bd63"
format_stamp: "Formatted at 2026-09-29 18:01:58 on dist-test-slave-r7ph"
server_key: "cdc76b74067a994735c89472349ddf47"
server_key_iv: "6e6c5160985b5bf19dcb26b32bdc69a1"
server_key_version: "encryptionkey@0"
I20260929 18:01:59.019027 25409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:01:59.034505 25409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:01:59.035311 25409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:01:59.035542 25409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:01:59.036183 25409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:01:59.037263 25409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:01:59.037339 25409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:59.037415 25409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:01:59.037467 25409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:59.047797 25409 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:38671
I20260929 18:01:59.047822 25536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:38671 every 8 connection(s)
I20260929 18:01:59.048941 25409 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 18:01:59.052172 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 25409
I20260929 18:01:59.052350 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance
I20260929 18:01:59.052783 7640 external_mini_cluster.cc:1428] Setting key e7ed415e2c50b36d1fe2be581eb7f56d
I20260929 18:01:59.055105 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:0
--local_ip_for_outbound_sockets=127.7.118.3
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:35567
--builtin_ntp_servers=127.7.118.20:36753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:01:59.060897 25257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:59.051202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:38103 (local address 127.7.118.62:35567)
0929 18:01:59.051361 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:59.051366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:59.052698 (+ 1332us) server_negotiation.cc:407] Connection header received
0929 18:01:59.054487 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:59.054491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:59.054549 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:59.054677 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:01:59.055827 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.056728 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.058380 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.058625 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.058722 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:59.060262 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:01:59.060265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:01:59.060268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:01:59.060283 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:01:59.060305 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:01:59.060314 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:01:59.060381 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:59.060656 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 18:01:59.060728 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 18:01:59.062465 25539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:59.051701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:35567 (local address 127.7.118.2:38103)
0929 18:01:59.052505 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:59.052562 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:59.054171 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:59.054851 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:01:59.054858 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:01:59.054965 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:01:59.055674 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:59.055690 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.056884 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.056887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:59.058189 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:59.058207 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.058812 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.058819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:59.059073 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:59.059857 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:01:59.059861 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:01:59.059931 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:01:59.060042 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:01:59.060419 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:01:59.060431 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:01:59.060441 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 18:01:59.060610 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 18:01:59.061002 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":171,"threads_started":1}
I20260929 18:01:59.064309 25537 heartbeater.cc:347] Connected to a master server at 127.7.118.62:35567
I20260929 18:01:59.064684 25537 heartbeater.cc:464] Registering TS with master...
I20260929 18:01:59.065348 25537 heartbeater.cc:511] Master 127.7.118.62:35567 requested a full tablet report, sending...
I20260929 18:01:59.066766 25218 ts_manager.cc:194] Registered new tserver with Master: f523d91e01b94918a2ed5b43a323bd63 (127.7.118.2:38671)
I20260929 18:01:59.067401 25218 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.118.2:38103
W20260929 18:01:59.178632 25540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:01:59.178949 25540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:01:59.178992 25540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:01:59.183912 25540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:01:59.184010 25540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:01:59.184073 25540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 18:01:59.189719 25540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:36753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:35567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:01:59.191051 25540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:01:59.192584 25540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:01:59.203640 25545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:01:59.203640 25548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:01:59.203727 25546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:01:59.203935 25540 server_base.cc:1034] running on GCE node
I20260929 18:01:59.204555 25540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:01:59.205428 25540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:01:59.206664 25540 hybrid_clock.cc:648] HybridClock initialized: now 1790704919206650 us; error 36 us; skew 500 ppm
I20260929 18:01:59.209081 25540 webserver.cc:460] Webserver started at http://127.7.118.3:39595/ using document root <none> and password file <none>
I20260929 18:01:59.209818 25540 fs_manager.cc:359] Metadata directory not provided
I20260929 18:01:59.209935 25540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:01:59.210223 25540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:01:59.212766 25540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-2/data/instance:
uuid: "74861c0b294c459aa3f811388706c2b4"
format_stamp: "Formatted at 2026-09-29 18:01:59 on dist-test-slave-r7ph"
server_key: "3e74d35d431a340ecd294817763bd615"
server_key_iv: "468bdce39b08f5b56a07061ac67b071c"
server_key_version: "encryptionkey@0"
I20260929 18:01:59.213469 25540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance:
uuid: "74861c0b294c459aa3f811388706c2b4"
format_stamp: "Formatted at 2026-09-29 18:01:59 on dist-test-slave-r7ph"
server_key: "3e74d35d431a340ecd294817763bd615"
server_key_iv: "468bdce39b08f5b56a07061ac67b071c"
server_key_version: "encryptionkey@0"
I20260929 18:01:59.218062 25540 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 18:01:59.221311 25554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:59.223112 25540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:01:59.223280 25540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "74861c0b294c459aa3f811388706c2b4"
format_stamp: "Formatted at 2026-09-29 18:01:59 on dist-test-slave-r7ph"
server_key: "3e74d35d431a340ecd294817763bd615"
server_key_iv: "468bdce39b08f5b56a07061ac67b071c"
server_key_version: "encryptionkey@0"
I20260929 18:01:59.223434 25540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:01:59.252579 25540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:01:59.253434 25540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:01:59.253670 25540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:01:59.254460 25540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:01:59.255566 25540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:01:59.255618 25540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:59.255693 25540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:01:59.255728 25540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:01:59.266104 25540 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:34781
I20260929 18:01:59.266129 25667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:34781 every 8 connection(s)
I20260929 18:01:59.267230 25540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 18:01:59.274358 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 25540
I20260929 18:01:59.274482 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance
I20260929 18:01:59.274799 7640 external_mini_cluster.cc:1428] Setting key 145ef97769301e24e703623d5c11fc3f
I20260929 18:01:59.278365 25257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:59.269572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:32797 (local address 127.7.118.62:35567)
0929 18:01:59.269743 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:59.269748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:59.271092 (+ 1344us) server_negotiation.cc:407] Connection header received
0929 18:01:59.272679 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:59.272682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:59.272747 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:59.272833 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:01:59.273853 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.274511 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.276097 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.276395 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.276528 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:59.277808 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:01:59.277812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:01:59.277814 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:01:59.277826 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:01:59.277847 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:01:59.277856 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:01:59.277917 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:59.278118 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 18:01:59.278177 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 18:01:59.279436 25670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:59.270045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:35567 (local address 127.7.118.3:32797)
0929 18:01:59.270891 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:59.270939 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:59.272398 (+ 1459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:59.273042 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:01:59.273050 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:01:59.273153 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:01:59.273687 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:59.273701 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.274668 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.274671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:59.275964 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:59.275975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.276566 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.276570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:59.276759 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:59.277516 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:01:59.277518 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:01:59.277566 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:01:59.277653 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:01:59.277966 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:01:59.277975 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:01:59.277981 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:01:59.278111 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 18:01:59.278549 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":219,"threads_started":1}
I20260929 18:01:59.280791 25668 heartbeater.cc:347] Connected to a master server at 127.7.118.62:35567
I20260929 18:01:59.281081 25668 heartbeater.cc:464] Registering TS with master...
I20260929 18:01:59.281692 25668 heartbeater.cc:511] Master 127.7.118.62:35567 requested a full tablet report, sending...
I20260929 18:01:59.282922 25218 ts_manager.cc:194] Registered new tserver with Master: 74861c0b294c459aa3f811388706c2b4 (127.7.118.3:34781)
I20260929 18:01:59.283571 25218 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.118.3:32797
I20260929 18:01:59.289940 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:01:59.296558 25257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:59.292448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35740 (local address 127.7.118.62:35567)
0929 18:01:59.292639 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:59.292644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:59.292778 (+ 134us) server_negotiation.cc:407] Connection header received
0929 18:01:59.292952 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:59.292957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:59.293025 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:59.293114 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:01:59.293416 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.293947 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.294843 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.295079 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.295196 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:59.296083 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:01:59.296087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:01:59.296089 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:01:59.296105 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:01:59.296130 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:01:59.296138 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:01:59.296214 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:59.296340 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 18:01:59.296404 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 18:01:59.300117 25218 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:01:59.303459 25218 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:01:59.320072 25682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:59.313808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48402 (local address 127.7.118.3:34781)
0929 18:01:59.315306 (+ 1498us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:59.315323 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:59.315359 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:01:59.315441 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:59.315451 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:59.315664 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:59.315832 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:01:59.316066 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.317024 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.318461 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.319252 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.319374 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:59.319609 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:59.319753 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 18:01:59.319867 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1388,"thread_start_us":173,"threads_started":1}
I20260929 18:01:59.320065 25683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:59.313645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:34781 (local address 127.0.0.1:48402)
0929 18:01:59.314628 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:59.314644 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:59.314742 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:59.315820 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:01:59.315823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:01:59.315842 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:01:59.315929 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:59.315934 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.317142 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.317145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:59.318339 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:59.318349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.319395 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.319399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:59.319528 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:59.319576 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 18:01:59.319721 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 18:01:59.319877 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":2,"thread_start_us":105,"threads_started":1}
I20260929 18:01:59.320493 25681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:59.313442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:38671 (local address 127.0.0.1:51202)
0929 18:01:59.314062 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:59.314117 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:59.314285 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:59.315156 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:01:59.315165 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:01:59.315194 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:01:59.315356 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:59.315364 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.316720 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.316724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:59.318566 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:59.318577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.319837 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.319840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:59.319925 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:59.319951 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:01:59.320036 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 18:01:59.320123 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":144,"threads_started":1}
I20260929 18:01:59.321091 25684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:59.314479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48748 (local address 127.7.118.1:35665)
0929 18:01:59.314986 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:59.315000 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:59.315026 (+ 26us) server_negotiation.cc:407] Connection header received
0929 18:01:59.315553 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:59.315559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:59.315723 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:59.315875 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:01:59.316247 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.317596 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.319046 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.319872 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.319986 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:59.320191 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:59.320693 (+ 502us) server_negotiation.cc:299] Negotiation successful
0929 18:01:59.320902 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":125,"threads_started":1}
I20260929 18:01:59.321192 25680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:59.313539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51202 (local address 127.7.118.2:38671)
0929 18:01:59.314595 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:59.314610 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:59.314638 (+ 28us) server_negotiation.cc:407] Connection header received
0929 18:01:59.314715 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:59.314721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:59.314944 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:59.315155 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:01:59.315486 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.316591 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.318707 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.319491 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.319593 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:59.319774 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:59.320923 (+ 1149us) server_negotiation.cc:299] Negotiation successful
0929 18:01:59.321042 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":942,"thread_start_us":98,"threads_started":1}
I20260929 18:01:59.321931 25679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:59.314299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:35665 (local address 127.0.0.1:48748)
0929 18:01:59.314953 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:59.314968 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:59.315052 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:59.316007 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:01:59.316010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:01:59.316029 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:01:59.316116 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:59.316121 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.317737 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.317740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:59.318915 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:59.318923 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.320274 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.320281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:59.320418 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:59.320444 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:01:59.320560 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 18:01:59.321754 (+ 1194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":92,"threads_started":1}
I20260929 18:01:59.323879 25471 tablet_service.cc:1467] Processing CreateTablet for tablet db4f9ece4a5b4773b100c0b6fa6c0c6a (DEFAULT_TABLE table=test-table [id=27556eacc41a4926b4f36f8464d11882]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:01:59.323776 25602 tablet_service.cc:1467] Processing CreateTablet for tablet db4f9ece4a5b4773b100c0b6fa6c0c6a (DEFAULT_TABLE table=test-table [id=27556eacc41a4926b4f36f8464d11882]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:01:59.324997 25602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db4f9ece4a5b4773b100c0b6fa6c0c6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:01:59.325004 25471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db4f9ece4a5b4773b100c0b6fa6c0c6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:01:59.324714 25340 tablet_service.cc:1467] Processing CreateTablet for tablet db4f9ece4a5b4773b100c0b6fa6c0c6a (DEFAULT_TABLE table=test-table [id=27556eacc41a4926b4f36f8464d11882]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:01:59.325907 25340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db4f9ece4a5b4773b100c0b6fa6c0c6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:01:59.332888 25685 tablet_bootstrap.cc:492] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4: Bootstrap starting.
I20260929 18:01:59.334414 25686 tablet_bootstrap.cc:492] T db4f9ece4a5b4773b100c0b6fa6c0c6a P f523d91e01b94918a2ed5b43a323bd63: Bootstrap starting.
I20260929 18:01:59.334587 25687 tablet_bootstrap.cc:492] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 63c04688e9d34e0386d15bcbf92316d5: Bootstrap starting.
I20260929 18:01:59.335424 25685 tablet_bootstrap.cc:654] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4: Neither blocks nor log segments found. Creating new log.
I20260929 18:01:59.336308 25685 log.cc:824] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4: Log is configured to *not* fsync() on all Append() calls
I20260929 18:01:59.336663 25686 tablet_bootstrap.cc:654] T db4f9ece4a5b4773b100c0b6fa6c0c6a P f523d91e01b94918a2ed5b43a323bd63: Neither blocks nor log segments found. Creating new log.
I20260929 18:01:59.336663 25687 tablet_bootstrap.cc:654] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 63c04688e9d34e0386d15bcbf92316d5: Neither blocks nor log segments found. Creating new log.
I20260929 18:01:59.337520 25686 log.cc:824] T db4f9ece4a5b4773b100c0b6fa6c0c6a P f523d91e01b94918a2ed5b43a323bd63: Log is configured to *not* fsync() on all Append() calls
I20260929 18:01:59.337522 25687 log.cc:824] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 63c04688e9d34e0386d15bcbf92316d5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:01:59.339810 25687 tablet_bootstrap.cc:492] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 63c04688e9d34e0386d15bcbf92316d5: No bootstrap required, opened a new log
I20260929 18:01:59.339810 25685 tablet_bootstrap.cc:492] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4: No bootstrap required, opened a new log
I20260929 18:01:59.339901 25686 tablet_bootstrap.cc:492] T db4f9ece4a5b4773b100c0b6fa6c0c6a P f523d91e01b94918a2ed5b43a323bd63: No bootstrap required, opened a new log
I20260929 18:01:59.340030 25687 ts_tablet_manager.cc:1397] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 63c04688e9d34e0386d15bcbf92316d5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:01:59.340030 25685 ts_tablet_manager.cc:1397] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 18:01:59.340162 25686 ts_tablet_manager.cc:1397] T db4f9ece4a5b4773b100c0b6fa6c0c6a P f523d91e01b94918a2ed5b43a323bd63: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:01:59.344391 25686 raft_consensus.cc:348] T db4f9ece4a5b4773b100c0b6fa6c0c6a P f523d91e01b94918a2ed5b43a323bd63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f523d91e01b94918a2ed5b43a323bd63" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 38671 } } peers { permanent_uuid: "63c04688e9d34e0386d15bcbf92316d5" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35665 } } peers { permanent_uuid: "74861c0b294c459aa3f811388706c2b4" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34781 } }
I20260929 18:01:59.344547 25685 raft_consensus.cc:348] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f523d91e01b94918a2ed5b43a323bd63" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 38671 } } peers { permanent_uuid: "63c04688e9d34e0386d15bcbf92316d5" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35665 } } peers { permanent_uuid: "74861c0b294c459aa3f811388706c2b4" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34781 } }
I20260929 18:01:59.344547 25687 raft_consensus.cc:348] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 63c04688e9d34e0386d15bcbf92316d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f523d91e01b94918a2ed5b43a323bd63" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 38671 } } peers { permanent_uuid: "63c04688e9d34e0386d15bcbf92316d5" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35665 } } peers { permanent_uuid: "74861c0b294c459aa3f811388706c2b4" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34781 } }
I20260929 18:01:59.344784 25686 raft_consensus.cc:374] T db4f9ece4a5b4773b100c0b6fa6c0c6a P f523d91e01b94918a2ed5b43a323bd63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:01:59.344825 25687 raft_consensus.cc:374] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 63c04688e9d34e0386d15bcbf92316d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:01:59.344825 25685 raft_consensus.cc:374] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:01:59.344858 25686 raft_consensus.cc:729] T db4f9ece4a5b4773b100c0b6fa6c0c6a P f523d91e01b94918a2ed5b43a323bd63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f523d91e01b94918a2ed5b43a323bd63, State: Initialized, Role: FOLLOWER
I20260929 18:01:59.344877 25685 raft_consensus.cc:729] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74861c0b294c459aa3f811388706c2b4, State: Initialized, Role: FOLLOWER
I20260929 18:01:59.344889 25687 raft_consensus.cc:729] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 63c04688e9d34e0386d15bcbf92316d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c04688e9d34e0386d15bcbf92316d5, State: Initialized, Role: FOLLOWER
I20260929 18:01:59.345397 25686 consensus_queue.cc:260] T db4f9ece4a5b4773b100c0b6fa6c0c6a P f523d91e01b94918a2ed5b43a323bd63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f523d91e01b94918a2ed5b43a323bd63" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 38671 } } peers { permanent_uuid: "63c04688e9d34e0386d15bcbf92316d5" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35665 } } peers { permanent_uuid: "74861c0b294c459aa3f811388706c2b4" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34781 } }
I20260929 18:01:59.345389 25685 consensus_queue.cc:260] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f523d91e01b94918a2ed5b43a323bd63" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 38671 } } peers { permanent_uuid: "63c04688e9d34e0386d15bcbf92316d5" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35665 } } peers { permanent_uuid: "74861c0b294c459aa3f811388706c2b4" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34781 } }
I20260929 18:01:59.345389 25687 consensus_queue.cc:260] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 63c04688e9d34e0386d15bcbf92316d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f523d91e01b94918a2ed5b43a323bd63" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 38671 } } peers { permanent_uuid: "63c04688e9d34e0386d15bcbf92316d5" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35665 } } peers { permanent_uuid: "74861c0b294c459aa3f811388706c2b4" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34781 } }
I20260929 18:01:59.346609 25685 ts_tablet_manager.cc:1428] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260929 18:01:59.346628 25686 ts_tablet_manager.cc:1428] T db4f9ece4a5b4773b100c0b6fa6c0c6a P f523d91e01b94918a2ed5b43a323bd63: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 18:01:59.346670 25406 heartbeater.cc:503] Master 127.7.118.62:35567 was elected leader, sending a full tablet report...
I20260929 18:01:59.346711 25668 heartbeater.cc:503] Master 127.7.118.62:35567 was elected leader, sending a full tablet report...
I20260929 18:01:59.347697 25687 ts_tablet_manager.cc:1428] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 63c04688e9d34e0386d15bcbf92316d5: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260929 18:01:59.347820 25537 heartbeater.cc:503] Master 127.7.118.62:35567 was elected leader, sending a full tablet report...
W20260929 18:01:59.351315 25407 tablet.cc:2369] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 63c04688e9d34e0386d15bcbf92316d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:01:59.373075 25692 raft_consensus.cc:482] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:01:59.373342 25692 raft_consensus.cc:504] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f523d91e01b94918a2ed5b43a323bd63" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 38671 } } peers { permanent_uuid: "63c04688e9d34e0386d15bcbf92316d5" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35665 } } peers { permanent_uuid: "74861c0b294c459aa3f811388706c2b4" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34781 } }
I20260929 18:01:59.374706 25692 leader_election.cc:288] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f523d91e01b94918a2ed5b43a323bd63 (127.7.118.2:38671), 63c04688e9d34e0386d15bcbf92316d5 (127.7.118.1:35665)
I20260929 18:01:59.379421 25670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:59.375077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:38671 (local address 127.7.118.3:35565)
0929 18:01:59.375249 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:59.375268 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:59.375407 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:59.375792 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:01:59.375796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:01:59.375816 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:01:59.375990 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:59.376000 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.377115 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.377120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:59.378078 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:59.378089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.378882 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.378886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:59.379011 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:59.379032 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:01:59.379151 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 18:01:59.379236 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":24}
I20260929 18:01:59.379417 25680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:59.375192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:35565 (local address 127.7.118.2:38671)
0929 18:01:59.375365 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:59.375371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:59.375392 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:01:59.375556 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:59.375559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:59.375628 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:59.375722 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:01:59.376133 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.376992 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.378224 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.378723 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.378856 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:59.378923 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:59.379151 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 18:01:59.379236 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 18:01:59.379424 25684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:59.375576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:37495 (local address 127.7.118.1:35665)
0929 18:01:59.375694 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:59.375700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:59.375997 (+ 297us) server_negotiation.cc:407] Connection header received
0929 18:01:59.376211 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:59.376214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:59.376266 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:59.376339 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:01:59.376742 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.377246 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.378442 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.378836 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.378961 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:59.379018 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:59.379222 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 18:01:59.379296 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:01:59.379462 25694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:59.375452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:35665 (local address 127.7.118.3:37495)
0929 18:01:59.375901 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:59.375915 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:01:59.376079 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:01:59.376404 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:01:59.376407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:01:59.376428 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:01:59.376549 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:59.376556 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.377375 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.377386 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:59.378297 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:01:59.378312 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.378976 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.378980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:01:59.379083 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:59.379113 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 18:01:59.379204 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 18:01:59.379275 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":38,"thread_start_us":138,"threads_started":1}
I20260929 18:01:59.380594 25491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4f9ece4a5b4773b100c0b6fa6c0c6a" candidate_uuid: "74861c0b294c459aa3f811388706c2b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f523d91e01b94918a2ed5b43a323bd63" is_pre_election: true
I20260929 18:01:59.380573 25360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4f9ece4a5b4773b100c0b6fa6c0c6a" candidate_uuid: "74861c0b294c459aa3f811388706c2b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c04688e9d34e0386d15bcbf92316d5" is_pre_election: true
I20260929 18:01:59.380949 25491 raft_consensus.cc:2454] T db4f9ece4a5b4773b100c0b6fa6c0c6a P f523d91e01b94918a2ed5b43a323bd63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74861c0b294c459aa3f811388706c2b4 in term 0.
I20260929 18:01:59.380966 25360 raft_consensus.cc:2454] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 63c04688e9d34e0386d15bcbf92316d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74861c0b294c459aa3f811388706c2b4 in term 0.
I20260929 18:01:59.381532 25555 leader_election.cc:302] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74861c0b294c459aa3f811388706c2b4, f523d91e01b94918a2ed5b43a323bd63; no voters:
I20260929 18:01:59.381887 25692 raft_consensus.cc:2781] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:01:59.381984 25692 raft_consensus.cc:482] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:01:59.382074 25692 raft_consensus.cc:3037] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:01:59.383293 25692 raft_consensus.cc:504] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f523d91e01b94918a2ed5b43a323bd63" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 38671 } } peers { permanent_uuid: "63c04688e9d34e0386d15bcbf92316d5" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35665 } } peers { permanent_uuid: "74861c0b294c459aa3f811388706c2b4" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34781 } }
I20260929 18:01:59.383751 25692 leader_election.cc:288] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4 [CANDIDATE]: Term 1 election: Requested vote from peers f523d91e01b94918a2ed5b43a323bd63 (127.7.118.2:38671), 63c04688e9d34e0386d15bcbf92316d5 (127.7.118.1:35665)
I20260929 18:01:59.384331 25491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4f9ece4a5b4773b100c0b6fa6c0c6a" candidate_uuid: "74861c0b294c459aa3f811388706c2b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f523d91e01b94918a2ed5b43a323bd63"
I20260929 18:01:59.384371 25360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4f9ece4a5b4773b100c0b6fa6c0c6a" candidate_uuid: "74861c0b294c459aa3f811388706c2b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c04688e9d34e0386d15bcbf92316d5"
I20260929 18:01:59.384496 25491 raft_consensus.cc:3037] T db4f9ece4a5b4773b100c0b6fa6c0c6a P f523d91e01b94918a2ed5b43a323bd63 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:01:59.384533 25360 raft_consensus.cc:3037] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 63c04688e9d34e0386d15bcbf92316d5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:01:59.385659 25491 raft_consensus.cc:2454] T db4f9ece4a5b4773b100c0b6fa6c0c6a P f523d91e01b94918a2ed5b43a323bd63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74861c0b294c459aa3f811388706c2b4 in term 1.
I20260929 18:01:59.385681 25360 raft_consensus.cc:2454] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 63c04688e9d34e0386d15bcbf92316d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74861c0b294c459aa3f811388706c2b4 in term 1.
I20260929 18:01:59.386096 25555 leader_election.cc:302] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74861c0b294c459aa3f811388706c2b4, f523d91e01b94918a2ed5b43a323bd63; no voters:
I20260929 18:01:59.386368 25692 raft_consensus.cc:2781] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:01:59.386656 25692 raft_consensus.cc:686] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4 [term 1 LEADER]: Becoming Leader. State: Replica: 74861c0b294c459aa3f811388706c2b4, State: Running, Role: LEADER
I20260929 18:01:59.387068 25692 consensus_queue.cc:237] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f523d91e01b94918a2ed5b43a323bd63" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 38671 } } peers { permanent_uuid: "63c04688e9d34e0386d15bcbf92316d5" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35665 } } peers { permanent_uuid: "74861c0b294c459aa3f811388706c2b4" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34781 } }
I20260929 18:01:59.391000 25213 catalog_manager.cc:5495] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74861c0b294c459aa3f811388706c2b4 (127.7.118.3). New cstate: current_term: 1 leader_uuid: "74861c0b294c459aa3f811388706c2b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f523d91e01b94918a2ed5b43a323bd63" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 38671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c04688e9d34e0386d15bcbf92316d5" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74861c0b294c459aa3f811388706c2b4" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 34781 } health_report { overall_health: HEALTHY } } }
I20260929 18:01:59.421175 25682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:59.417478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48404 (local address 127.7.118.3:34781)
0929 18:01:59.417657 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 18:01:59.417662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:01:59.417682 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:01:59.417757 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:01:59.417759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:01:59.417810 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:01:59.417930 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:01:59.418194 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.418732 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.419588 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:01:59.419812 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:01:59.419926 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:01:59.420257 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:01:59.420699 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:01:59.420858 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 18:01:59.420935 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 18:01:59.420992 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 18:01:59.428025 25360 raft_consensus.cc:1261] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 63c04688e9d34e0386d15bcbf92316d5 [term 1 FOLLOWER]: Refusing update from remote peer 74861c0b294c459aa3f811388706c2b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:01:59.428025 25491 raft_consensus.cc:1261] T db4f9ece4a5b4773b100c0b6fa6c0c6a P f523d91e01b94918a2ed5b43a323bd63 [term 1 FOLLOWER]: Refusing update from remote peer 74861c0b294c459aa3f811388706c2b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:01:59.428908 25695 consensus_queue.cc:1035] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c04688e9d34e0386d15bcbf92316d5" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:01:59.429126 25692 consensus_queue.cc:1035] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f523d91e01b94918a2ed5b43a323bd63" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 38671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:01:59.447422 25218 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:01:59.447647 25218 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:01:59.450939 25218 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f6dc48850eec486fa7aac1ea84653b6f: Sending DeleteTablet for 3 replicas of tablet db4f9ece4a5b4773b100c0b6fa6c0c6a
I20260929 18:01:59.451970 25602 tablet_service.cc:1514] Processing DeleteTablet for tablet db4f9ece4a5b4773b100c0b6fa6c0c6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:01:59 UTC) from {username='slave'} at 127.0.0.1:48402
I20260929 18:01:59.451993 25340 tablet_service.cc:1514] Processing DeleteTablet for tablet db4f9ece4a5b4773b100c0b6fa6c0c6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:01:59 UTC) from {username='slave'} at 127.0.0.1:48748
I20260929 18:01:59.452453 25471 tablet_service.cc:1514] Processing DeleteTablet for tablet db4f9ece4a5b4773b100c0b6fa6c0c6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:01:59 UTC) from {username='slave'} at 127.0.0.1:51202
I20260929 18:01:59.452662 25709 tablet_replica.cc:330] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 63c04688e9d34e0386d15bcbf92316d5: stopping tablet replica
I20260929 18:01:59.452970 25710 tablet_replica.cc:330] T db4f9ece4a5b4773b100c0b6fa6c0c6a P f523d91e01b94918a2ed5b43a323bd63: stopping tablet replica
I20260929 18:01:59.453120 25709 raft_consensus.cc:2229] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 63c04688e9d34e0386d15bcbf92316d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:01:59.453339 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 25278
I20260929 18:01:59.454128 25710 raft_consensus.cc:2229] T db4f9ece4a5b4773b100c0b6fa6c0c6a P f523d91e01b94918a2ed5b43a323bd63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:01:59.454622 25710 raft_consensus.cc:2258] T db4f9ece4a5b4773b100c0b6fa6c0c6a P f523d91e01b94918a2ed5b43a323bd63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:01:59.455153 25708 tablet_replica.cc:330] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4: stopping tablet replica
I20260929 18:01:59.455510 25708 raft_consensus.cc:2229] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:01:59.455925 25708 raft_consensus.cc:2258] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:01:59.456839 25710 ts_tablet_manager.cc:1905] T db4f9ece4a5b4773b100c0b6fa6c0c6a P f523d91e01b94918a2ed5b43a323bd63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:01:59.457490 25708 ts_tablet_manager.cc:1905] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:01:59.460076 25710 ts_tablet_manager.cc:1918] T db4f9ece4a5b4773b100c0b6fa6c0c6a P f523d91e01b94918a2ed5b43a323bd63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:01:59.460292 25710 log.cc:1196] T db4f9ece4a5b4773b100c0b6fa6c0c6a P f523d91e01b94918a2ed5b43a323bd63: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-1/wal/wals/db4f9ece4a5b4773b100c0b6fa6c0c6a
I20260929 18:01:59.460709 25710 ts_tablet_manager.cc:1939] T db4f9ece4a5b4773b100c0b6fa6c0c6a P f523d91e01b94918a2ed5b43a323bd63: Deleting consensus metadata
I20260929 18:01:59.461315 25708 ts_tablet_manager.cc:1918] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:01:59.461442 25708 log.cc:1196] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704657566865-7640-0/minicluster-data/ts-2/wal/wals/db4f9ece4a5b4773b100c0b6fa6c0c6a
I20260929 18:01:59.461802 25708 ts_tablet_manager.cc:1939] T db4f9ece4a5b4773b100c0b6fa6c0c6a P 74861c0b294c459aa3f811388706c2b4: Deleting consensus metadata
I20260929 18:01:59.462589 25202 catalog_manager.cc:4841] TS f523d91e01b94918a2ed5b43a323bd63 (127.7.118.2:38671): tablet db4f9ece4a5b4773b100c0b6fa6c0c6a (table test-table [id=27556eacc41a4926b4f36f8464d11882]) successfully deleted
I20260929 18:01:59.463263 25205 catalog_manager.cc:4841] TS 74861c0b294c459aa3f811388706c2b4 (127.7.118.3:34781): tablet db4f9ece4a5b4773b100c0b6fa6c0c6a (table test-table [id=27556eacc41a4926b4f36f8464d11882]) successfully deleted
W20260929 18:01:59.463799 25202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.118.1:35665 (error 108)
I20260929 18:01:59.464763 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 25409
I20260929 18:01:59.464921 25679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:01:59.464619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:35665 (local address 127.0.0.1:48756)
0929 18:01:59.464739 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 18:01:59.464811 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.1:35665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 18:01:59.465093 25202 catalog_manager.cc:4584] TS 63c04688e9d34e0386d15bcbf92316d5 (127.7.118.1:35665): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet db4f9ece4a5b4773b100c0b6fa6c0c6a: Network error: Client connection negotiation failed: client connection to 127.7.118.1:35665: connect: Connection refused (error 111)
I20260929 18:01:59.471493 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 25540
I20260929 18:01:59.476691 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 25187
2026-09-29T18:01:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1145 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:01:59 dist-test-slave-r7ph krb5kdc[25714](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:01:59 dist-test-slave-r7ph krb5kdc[25714](info): set up 2 sockets
Sep 29 18:01:59 dist-test-slave-r7ph krb5kdc[25714](info): commencing operation
krb5kdc: starting...
W20260929 18:02:00.554370 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 18:02:00 dist-test-slave-r7ph krb5kdc[25714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:02:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:02:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 18:02:00.694795 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:38479
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:35061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:38479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:00.812870 25727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:00.813195 25727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:00.813239 25727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:00.817662 25727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:00.817750 25727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:00.817771 25727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:00.817788 25727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:00.817804 25727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:00.823635 25727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:35061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:38479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:38479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:00.824832 25727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:00.825970 25727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:00.835845 25732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:00.835830 25733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:00.835871 25727 server_base.cc:1034] running on GCE node
W20260929 18:02:00.835830 25735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:00.836802 25727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:00.838171 25727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:00.839386 25727 hybrid_clock.cc:648] HybridClock initialized: now 1790704920839365 us; error 35 us; skew 500 ppm
Sep 29 18:02:00 dist-test-slave-r7ph krb5kdc[25714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704920, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:00.842448 25727 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 18:02:00.843385 25727 webserver.cc:460] Webserver started at http://127.7.118.62:36759/ using document root <none> and password file <none>
I20260929 18:02:00.843950 25727 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:00.844013 25727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:00.844183 25727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:00.846386 25727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "5468f7532f41442685f2466829c31a83"
format_stamp: "Formatted at 2026-09-29 18:02:00 on dist-test-slave-r7ph"
server_key: "d2bcbf6cbde432b385eb211fa635cd43"
server_key_iv: "0e35748753443f35f8a28dfeafaac8da"
server_key_version: "encryptionkey@0"
I20260929 18:02:00.846966 25727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "5468f7532f41442685f2466829c31a83"
format_stamp: "Formatted at 2026-09-29 18:02:00 on dist-test-slave-r7ph"
server_key: "d2bcbf6cbde432b385eb211fa635cd43"
server_key_iv: "0e35748753443f35f8a28dfeafaac8da"
server_key_version: "encryptionkey@0"
I20260929 18:02:00.851131 25727 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 18:02:00.853955 25742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:00.855490 25727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:02:00.855660 25727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "5468f7532f41442685f2466829c31a83"
format_stamp: "Formatted at 2026-09-29 18:02:00 on dist-test-slave-r7ph"
server_key: "d2bcbf6cbde432b385eb211fa635cd43"
server_key_iv: "0e35748753443f35f8a28dfeafaac8da"
server_key_version: "encryptionkey@0"
I20260929 18:02:00.855803 25727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:00.885962 25727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:00.889657 25727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:00.889912 25727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:00.897444 25727 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:38479
I20260929 18:02:00.897476 25794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:38479 every 8 connection(s)
I20260929 18:02:00.898619 25727 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:02:00.901623 25795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:00.902416 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 25727
I20260929 18:02:00.902568 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:02:00.902859 7640 external_mini_cluster.cc:1428] Setting key f896954697ce1899afc10b358c1fe769
I20260929 18:02:00.908087 25795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5468f7532f41442685f2466829c31a83: Bootstrap starting.
I20260929 18:02:00.910480 25795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5468f7532f41442685f2466829c31a83: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:00.911229 25795 log.cc:824] T 00000000000000000000000000000000 P 5468f7532f41442685f2466829c31a83: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:02:00 dist-test-slave-r7ph krb5kdc[25714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:02:00.913549 25795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5468f7532f41442685f2466829c31a83: No bootstrap required, opened a new log
I20260929 18:02:00.917090 25795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5468f7532f41442685f2466829c31a83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5468f7532f41442685f2466829c31a83" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 38479 } }
I20260929 18:02:00.917366 25795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5468f7532f41442685f2466829c31a83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:00.917430 25795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5468f7532f41442685f2466829c31a83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5468f7532f41442685f2466829c31a83, State: Initialized, Role: FOLLOWER
I20260929 18:02:00.917954 25795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5468f7532f41442685f2466829c31a83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5468f7532f41442685f2466829c31a83" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 38479 } }
I20260929 18:02:00.918087 25795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5468f7532f41442685f2466829c31a83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:02:00.918170 25795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5468f7532f41442685f2466829c31a83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:02:00.918313 25795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5468f7532f41442685f2466829c31a83 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:00.918507 25798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:00.904494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55084 (local address 127.7.118.62:38479)
0929 18:02:00.905043 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:00.905060 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:00.905100 (+ 40us) server_negotiation.cc:407] Connection header received
0929 18:02:00.905803 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:00.905821 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:00.906180 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:00.906646 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:00.907554 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:00.908604 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:00.909478 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:00.909985 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:00.910160 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:00.913273 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:00.913279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:00.913295 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:00.913327 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:00.915311 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:00.915824 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:00.915829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:00.915834 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:00.915953 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:00.916253 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:00.916257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:00.916260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:00.916679 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:00.916905 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:00.917156 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 18:02:00.917504 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":150,"threads_started":1}
I20260929 18:02:00.919359 25795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5468f7532f41442685f2466829c31a83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5468f7532f41442685f2466829c31a83" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 38479 } }
I20260929 18:02:00.919732 25795 leader_election.cc:302] T 00000000000000000000000000000000 P 5468f7532f41442685f2466829c31a83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5468f7532f41442685f2466829c31a83; no voters:
I20260929 18:02:00.920086 25795 leader_election.cc:288] T 00000000000000000000000000000000 P 5468f7532f41442685f2466829c31a83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:02:00.920219 25800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5468f7532f41442685f2466829c31a83 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:00.920408 25800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5468f7532f41442685f2466829c31a83 [term 1 LEADER]: Becoming Leader. State: Replica: 5468f7532f41442685f2466829c31a83, State: Running, Role: LEADER
I20260929 18:02:00.920761 25800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5468f7532f41442685f2466829c31a83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5468f7532f41442685f2466829c31a83" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 38479 } }
I20260929 18:02:00.921412 25795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5468f7532f41442685f2466829c31a83 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:02:00.922863 25802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5468f7532f41442685f2466829c31a83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5468f7532f41442685f2466829c31a83. Latest consensus state: current_term: 1 leader_uuid: "5468f7532f41442685f2466829c31a83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5468f7532f41442685f2466829c31a83" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 38479 } } }
I20260929 18:02:00.922897 25801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5468f7532f41442685f2466829c31a83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5468f7532f41442685f2466829c31a83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5468f7532f41442685f2466829c31a83" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 38479 } } }
I20260929 18:02:00.923039 25802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5468f7532f41442685f2466829c31a83 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:00.923039 25801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5468f7532f41442685f2466829c31a83 [sys.catalog]: This master's current role is: LEADER
W20260929 18:02:00.926808 25815 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5468f7532f41442685f2466829c31a83: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:02:00.926891 25815 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:02:00.926971 25810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:02:00.927676 25810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:02:00.934139 25810 catalog_manager.cc:1347] Generated new cluster ID: a1c31e8635fe470996c413dd2c6690ff
I20260929 18:02:00.934239 25810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:02:00.940649 25810 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:02:00.941727 25810 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:02:00.953658 25810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5468f7532f41442685f2466829c31a83: Generated new TSK 0
I20260929 18:02:00.954512 25810 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.118.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
I20260929 18:02:00.990844 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:0
--local_ip_for_outbound_sockets=127.7.118.1
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:38479
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:35061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:01.108004 25821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:01.108302 25821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:01.108348 25821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:01.112536 25821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:01.112613 25821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:01.112676 25821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 18:02:01.118302 25821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:35061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:38479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:01.119390 25821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:01.120396 25821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:01.131774 25827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:01.131875 25821 server_base.cc:1034] running on GCE node
W20260929 18:02:01.131749 25829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:01.131749 25826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:01.132448 25821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:01.133229 25821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:01.134465 25821 hybrid_clock.cc:648] HybridClock initialized: now 1790704921134441 us; error 42 us; skew 500 ppm
Sep 29 18:02:01 dist-test-slave-r7ph krb5kdc[25714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704921, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:01.137530 25821 init.cc:377] Logged in from keytab as kudu/127.7.118.1@KRBTEST.COM (short username kudu)
I20260929 18:02:01.138643 25821 webserver.cc:460] Webserver started at http://127.7.118.1:41425/ using document root <none> and password file <none>
I20260929 18:02:01.139264 25821 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:01.139355 25821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:01.139693 25821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:01.142033 25821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-0/data/instance:
uuid: "146b399202d745fa9e63ef7d05d67fc8"
format_stamp: "Formatted at 2026-09-29 18:02:01 on dist-test-slave-r7ph"
server_key: "5df735112f21cc93f5922273f0bd9997"
server_key_iv: "824c962516aa1765f28df3781032f508"
server_key_version: "encryptionkey@0"
I20260929 18:02:01.142810 25821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance:
uuid: "146b399202d745fa9e63ef7d05d67fc8"
format_stamp: "Formatted at 2026-09-29 18:02:01 on dist-test-slave-r7ph"
server_key: "5df735112f21cc93f5922273f0bd9997"
server_key_iv: "824c962516aa1765f28df3781032f508"
server_key_version: "encryptionkey@0"
I20260929 18:02:01.147132 25821 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:02:01.150147 25836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:01.151734 25821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:02:01.151889 25821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "146b399202d745fa9e63ef7d05d67fc8"
format_stamp: "Formatted at 2026-09-29 18:02:01 on dist-test-slave-r7ph"
server_key: "5df735112f21cc93f5922273f0bd9997"
server_key_iv: "824c962516aa1765f28df3781032f508"
server_key_version: "encryptionkey@0"
I20260929 18:02:01.152047 25821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:01.165766 25821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:01.169502 25821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:01.169727 25821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:01.170512 25821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:02:01.171432 25821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:02:01.171482 25821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:01.171548 25821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:02:01.171581 25821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:01.182827 25821 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:41301
I20260929 18:02:01.182842 25949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:41301 every 8 connection(s)
I20260929 18:02:01.183935 25821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 18:02:01.188084 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 25821
I20260929 18:02:01.188246 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance
I20260929 18:02:01.188545 7640 external_mini_cluster.cc:1428] Setting key 77dd1f3b050be6b9dfb80859da97b3bd
Sep 29 18:02:01 dist-test-slave-r7ph krb5kdc[25714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704921, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.118.2@KRBTEST.COM; defaulting to no policy
I20260929 18:02:01.199069 25798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:01.186034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:44577 (local address 127.7.118.62:38479)
0929 18:02:01.186180 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:01.186184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:01.187135 (+ 951us) server_negotiation.cc:407] Connection header received
0929 18:02:01.188174 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:01.188177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:01.188237 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:01.188326 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:01.189718 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.190341 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.191587 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.191824 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.191953 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:01.194960 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:01.194964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:01.194966 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:01.195001 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:01.196636 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:01.197231 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:01.197235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:01.197237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:01.197331 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:01.197754 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:01.197757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:01.197759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:01.197973 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:01.198086 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:01.198731 (+ 645us) server_negotiation.cc:299] Negotiation successful
0929 18:02:01.198891 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:02:01.199977 25952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:01.186429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:38479 (local address 127.7.118.1:44577)
0929 18:02:01.186945 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:01.186995 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:01.187952 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:01.188546 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:01.188553 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:01.188983 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:01.189508 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:01.189534 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.190483 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.190491 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:01.191434 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:01.191444 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.192003 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.192008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:01.192193 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:01.192864 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:02:01.192882 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:01.194732 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:01.196782 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:01.196790 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:01.196810 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:01.197117 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:01.197434 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:01.197438 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:01.197440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:01.197627 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:01.198084 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:01.198092 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:01.198470 (+ 378us) client_negotiation.cc:769] Sending connection context
0929 18:02:01.198681 (+ 211us) client_negotiation.cc:240] Negotiation successful
0929 18:02:01.199063 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":124,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.2@KRBTEST.COM" created.
I20260929 18:02:01.201619 25950 heartbeater.cc:347] Connected to a master server at 127.7.118.62:38479
I20260929 18:02:01.201930 25950 heartbeater.cc:464] Registering TS with master...
I20260929 18:02:01.202648 25950 heartbeater.cc:511] Master 127.7.118.62:38479 requested a full tablet report, sending...
I20260929 18:02:01.204675 25759 ts_manager.cc:194] Registered new tserver with Master: 146b399202d745fa9e63ef7d05d67fc8 (127.7.118.1:41301)
I20260929 18:02:01.206648 25759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.1@KRBTEST.COM'} at 127.7.118.1:44577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
I20260929 18:02:01.221777 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:0
--local_ip_for_outbound_sockets=127.7.118.2
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:38479
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:35061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:01.336826 25955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:01.337118 25955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:01.337163 25955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:01.341372 25955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:01.341451 25955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:01.341513 25955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 18:02:01.347060 25955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:35061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:38479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:01.348167 25955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:01.349177 25955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:01.359778 25960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:01.359923 25963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:01.359931 25961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:01.360016 25955 server_base.cc:1034] running on GCE node
I20260929 18:02:01.360702 25955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:01.361460 25955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:01.362699 25955 hybrid_clock.cc:648] HybridClock initialized: now 1790704921362677 us; error 35 us; skew 500 ppm
Sep 29 18:02:01 dist-test-slave-r7ph krb5kdc[25714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704921, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:01.365810 25955 init.cc:377] Logged in from keytab as kudu/127.7.118.2@KRBTEST.COM (short username kudu)
I20260929 18:02:01.366925 25955 webserver.cc:460] Webserver started at http://127.7.118.2:40087/ using document root <none> and password file <none>
I20260929 18:02:01.367538 25955 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:01.367636 25955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:01.367882 25955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:01.370167 25955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-1/data/instance:
uuid: "ffd8fe4614e5415f9512abb623d1c823"
format_stamp: "Formatted at 2026-09-29 18:02:01 on dist-test-slave-r7ph"
server_key: "f0e3fa286d8ff048270bfd3fa2185345"
server_key_iv: "16fdb9971bd4bdc025ee38d87d919642"
server_key_version: "encryptionkey@0"
I20260929 18:02:01.370920 25955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance:
uuid: "ffd8fe4614e5415f9512abb623d1c823"
format_stamp: "Formatted at 2026-09-29 18:02:01 on dist-test-slave-r7ph"
server_key: "f0e3fa286d8ff048270bfd3fa2185345"
server_key_iv: "16fdb9971bd4bdc025ee38d87d919642"
server_key_version: "encryptionkey@0"
I20260929 18:02:01.375131 25955 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:02:01.377871 25970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:01.379326 25955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:02:01.379480 25955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "ffd8fe4614e5415f9512abb623d1c823"
format_stamp: "Formatted at 2026-09-29 18:02:01 on dist-test-slave-r7ph"
server_key: "f0e3fa286d8ff048270bfd3fa2185345"
server_key_iv: "16fdb9971bd4bdc025ee38d87d919642"
server_key_version: "encryptionkey@0"
I20260929 18:02:01.379591 25955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:01.431440 25955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:01.435209 25955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:01.435376 25955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:01.435972 25955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:02:01.436797 25955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:02:01.436847 25955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:01.436889 25955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:02:01.436904 25955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:01.446142 25955 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:40057
I20260929 18:02:01.446166 26083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:40057 every 8 connection(s)
I20260929 18:02:01.447294 25955 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 18:02:01.449775 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 25955
I20260929 18:02:01.449921 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance
I20260929 18:02:01.450295 7640 external_mini_cluster.cc:1428] Setting key dac9d00247a5da620d21d7158832796f
Sep 29 18:02:01 dist-test-slave-r7ph krb5kdc[25714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704921, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.118.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.3@KRBTEST.COM" created.
I20260929 18:02:01.464723 25798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:01.449513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:55027 (local address 127.7.118.62:38479)
0929 18:02:01.449663 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:01.449667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:01.450927 (+ 1260us) server_negotiation.cc:407] Connection header received
0929 18:02:01.452545 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:01.452549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:01.452607 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:01.452683 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:01.454367 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.455356 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.456889 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.457395 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.457551 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:01.460272 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:01.460280 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:01.460286 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:01.460327 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:01.461990 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:01.462690 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:01.462694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:01.462695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:01.462789 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:01.463377 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:01.463381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:01.463382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:01.463601 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:01.463687 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:01.464445 (+ 758us) server_negotiation.cc:299] Negotiation successful
0929 18:02:01.464591 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:02:01.466019 26086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:01.449975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:38479 (local address 127.7.118.2:55027)
0929 18:02:01.450692 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:01.450767 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:01.452299 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:01.452964 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:01.452978 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:01.453531 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:01.454161 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:01.454178 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.455519 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.455522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:01.456650 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:01.456661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.457609 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.457612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:01.457777 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:01.458471 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:02:01.458493 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:01.460067 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:01.462108 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:01.462115 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:01.462131 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:01.462539 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:01.462998 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:01.463005 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:01.463009 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:01.463226 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:01.463735 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:01.463749 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:01.464115 (+ 366us) client_negotiation.cc:769] Sending connection context
0929 18:02:01.464343 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 18:02:01.464780 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":232,"threads_started":1}
I20260929 18:02:01.467478 26084 heartbeater.cc:347] Connected to a master server at 127.7.118.62:38479
I20260929 18:02:01.467849 26084 heartbeater.cc:464] Registering TS with master...
I20260929 18:02:01.468559 26084 heartbeater.cc:511] Master 127.7.118.62:38479 requested a full tablet report, sending...
I20260929 18:02:01.469772 25759 ts_manager.cc:194] Registered new tserver with Master: ffd8fe4614e5415f9512abb623d1c823 (127.7.118.2:40057)
I20260929 18:02:01.470760 25759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.2@KRBTEST.COM'} at 127.7.118.2:55027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
I20260929 18:02:01.482417 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:0
--local_ip_for_outbound_sockets=127.7.118.3
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:38479
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:35061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:01.603541 26089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:01.603847 26089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:01.603894 26089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:01.608191 26089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:01.608287 26089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:01.608417 26089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 18:02:01.614108 26089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:35061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:38479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:01.615316 26089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:01.616401 26089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:01.627552 26097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:01.627552 26095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:01.627744 26094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:01.627929 26089 server_base.cc:1034] running on GCE node
I20260929 18:02:01.628473 26089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:01.629256 26089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:01.630434 26089 hybrid_clock.cc:648] HybridClock initialized: now 1790704921630421 us; error 33 us; skew 500 ppm
Sep 29 18:02:01 dist-test-slave-r7ph krb5kdc[25714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704921, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:01.633982 26089 init.cc:377] Logged in from keytab as kudu/127.7.118.3@KRBTEST.COM (short username kudu)
I20260929 18:02:01.635278 26089 webserver.cc:460] Webserver started at http://127.7.118.3:40469/ using document root <none> and password file <none>
I20260929 18:02:01.636008 26089 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:01.636085 26089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:01.636329 26089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:01.638805 26089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-2/data/instance:
uuid: "ab74e2725dd743c894908611a0085428"
format_stamp: "Formatted at 2026-09-29 18:02:01 on dist-test-slave-r7ph"
server_key: "cbdded954fa824bfcc11fa87f771ead6"
server_key_iv: "94d4b2a67ae4c50bc2df724c77a33216"
server_key_version: "encryptionkey@0"
I20260929 18:02:01.639559 26089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance:
uuid: "ab74e2725dd743c894908611a0085428"
format_stamp: "Formatted at 2026-09-29 18:02:01 on dist-test-slave-r7ph"
server_key: "cbdded954fa824bfcc11fa87f771ead6"
server_key_iv: "94d4b2a67ae4c50bc2df724c77a33216"
server_key_version: "encryptionkey@0"
I20260929 18:02:01.644181 26089 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:02:01.647480 26104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:01.649168 26089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:02:01.649335 26089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "ab74e2725dd743c894908611a0085428"
format_stamp: "Formatted at 2026-09-29 18:02:01 on dist-test-slave-r7ph"
server_key: "cbdded954fa824bfcc11fa87f771ead6"
server_key_iv: "94d4b2a67ae4c50bc2df724c77a33216"
server_key_version: "encryptionkey@0"
I20260929 18:02:01.649479 26089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:01.695639 26089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:01.699572 26089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:01.699846 26089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:01.700672 26089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:02:01.701608 26089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:02:01.701663 26089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:01.701704 26089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:02:01.701756 26089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:01.712823 26089 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:37459
I20260929 18:02:01.712844 26217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:37459 every 8 connection(s)
I20260929 18:02:01.713918 26089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 18:02:01.720708 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 26089
I20260929 18:02:01.720839 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance
I20260929 18:02:01.721242 7640 external_mini_cluster.cc:1428] Setting key e1f7c7bf65820e95e63bd0addd5bc0fc
Sep 29 18:02:01 dist-test-slave-r7ph krb5kdc[25714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704921, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:02:01.731681 25798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:01.716234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:33989 (local address 127.7.118.62:38479)
0929 18:02:01.716459 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:01.716463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:01.717462 (+ 999us) server_negotiation.cc:407] Connection header received
0929 18:02:01.719201 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:01.719205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:01.719271 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:01.719465 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:01.721151 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.722060 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.723810 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.724124 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.724223 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:01.727320 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:01.727323 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:01.727325 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:01.727355 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:01.729024 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:01.729720 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:01.729724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:01.729725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:01.729814 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:01.730458 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:01.730461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:01.730462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:01.730679 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:01.730771 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:01.731405 (+ 634us) server_negotiation.cc:299] Negotiation successful
0929 18:02:01.731543 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260929 18:02:01.732645 26220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:01.716740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:38479 (local address 127.7.118.3:33989)
0929 18:02:01.717246 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:01.717295 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:01.718842 (+ 1547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:01.719750 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:01.719762 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:01.720229 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:01.720898 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:01.720929 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.722328 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.722332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:01.723657 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:01.723671 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.724359 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.724364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:01.724552 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:01.725303 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:02:01.725327 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:01.727055 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:01.729186 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:01.729192 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:01.729209 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:01.729575 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:01.729955 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:01.729963 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:01.729965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:01.730203 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:01.730807 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:01.730814 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:01.731107 (+ 293us) client_negotiation.cc:769] Sending connection context
0929 18:02:01.731324 (+ 217us) client_negotiation.cc:240] Negotiation successful
0929 18:02:01.731643 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":130,"threads_started":1}
I20260929 18:02:01.734076 26218 heartbeater.cc:347] Connected to a master server at 127.7.118.62:38479
I20260929 18:02:01.734442 26218 heartbeater.cc:464] Registering TS with master...
I20260929 18:02:01.735055 26218 heartbeater.cc:511] Master 127.7.118.62:38479 requested a full tablet report, sending...
I20260929 18:02:01.736294 25758 ts_manager.cc:194] Registered new tserver with Master: ab74e2725dd743c894908611a0085428 (127.7.118.3:37459)
I20260929 18:02:01.737030 25758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.3@KRBTEST.COM'} at 127.7.118.3:33989
I20260929 18:02:01.749336 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:02:01.761025 25798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:01.751655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55088 (local address 127.7.118.62:38479)
0929 18:02:01.751805 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:01.751809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:01.751904 (+ 95us) server_negotiation.cc:407] Connection header received
0929 18:02:01.752093 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:01.752096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:01.752161 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:01.752248 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:01.753086 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.753659 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.754706 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.754969 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.755072 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:01.757161 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:01.757164 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:01.757166 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:01.757206 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:01.758862 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:01.759458 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:01.759461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:01.759462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:01.759577 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:01.759938 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:01.759940 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:01.759941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:01.760166 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:01.760351 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:01.760694 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 18:02:01.760827 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 18:02:01.764609 25758 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:02:01.768106 25758 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:02:01.784528 26230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:01.778037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:41301 (local address 127.0.0.1:46242)
0929 18:02:01.778640 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:01.778696 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:01.778837 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:01.779502 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:01.779508 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:01.779543 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:02:01.779723 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:01.779730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.781339 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.781342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:01.782734 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:01.782743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.784009 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.784012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:01.784125 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:01.784166 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 18:02:01.784270 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 18:02:01.784374 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":76,"thread_start_us":78,"threads_started":1}
I20260929 18:02:01.784603 26231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:01.778172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46242 (local address 127.7.118.1:41301)
0929 18:02:01.778975 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:01.778982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:01.779007 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:02:01.779090 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:01.779095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:01.779329 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:01.779898 (+ 569us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:02:01.780029 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.781203 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.782874 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.783848 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.784130 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:01.784254 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:01.784312 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 18:02:01.784421 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":195,"threads_started":1}
I20260929 18:02:01.784762 26233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:01.778064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:37459 (local address 127.0.0.1:47284)
0929 18:02:01.779079 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:01.779093 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:01.779170 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:01.779941 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:01.779945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:01.779964 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:02:01.780085 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:01.780091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.781461 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.781466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:01.782979 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:01.782990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.784324 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.784327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:01.784432 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:01.784455 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:02:01.784533 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 18:02:01.784605 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"mutex_wait_us":74,"thread_start_us":143,"threads_started":1}
I20260929 18:02:01.784914 26232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:01.778765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47284 (local address 127.7.118.3:37459)
0929 18:02:01.779169 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:01.779181 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:01.779204 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:02:01.779277 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:01.779282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:01.779587 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:01.779744 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:02:01.780224 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.781323 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.783123 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.783932 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.784031 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:01.784197 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:01.784594 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 18:02:01.784703 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":99,"threads_started":1}
I20260929 18:02:01.785379 26229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:01.778832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:40057 (local address 127.0.0.1:52384)
0929 18:02:01.779320 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:01.779331 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:01.779391 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:01.780469 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:01.780472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:01.780490 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:02:01.780579 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:01.780585 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.781597 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.781601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:01.782990 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:01.782996 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.783949 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.783953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:01.784065 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:01.785095 (+ 1030us) client_negotiation.cc:769] Sending connection context
0929 18:02:01.785177 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 18:02:01.785262 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":145,"thread_start_us":93,"threads_started":1}
I20260929 18:02:01.785848 26234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:01.779627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52384 (local address 127.7.118.2:40057)
0929 18:02:01.780053 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:01.780065 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:01.780086 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:02:01.780162 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:01.780170 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:01.780331 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:01.780490 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:02:01.780670 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.781473 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.783143 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.783816 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.783942 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:01.784111 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:01.785236 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0929 18:02:01.785344 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":158,"threads_started":1}
I20260929 18:02:01.787881 25884 tablet_service.cc:1467] Processing CreateTablet for tablet 05609d83481f48cca730afee0f272d9b (DEFAULT_TABLE table=test-table [id=850305b369b54bc6896053000a4c925a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:02:01.788172 26018 tablet_service.cc:1467] Processing CreateTablet for tablet 05609d83481f48cca730afee0f272d9b (DEFAULT_TABLE table=test-table [id=850305b369b54bc6896053000a4c925a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:02:01.788870 25884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05609d83481f48cca730afee0f272d9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:01.789160 26018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05609d83481f48cca730afee0f272d9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:01.792346 26152 tablet_service.cc:1467] Processing CreateTablet for tablet 05609d83481f48cca730afee0f272d9b (DEFAULT_TABLE table=test-table [id=850305b369b54bc6896053000a4c925a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:02:01.793578 26152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05609d83481f48cca730afee0f272d9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:01.796281 26236 tablet_bootstrap.cc:492] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8: Bootstrap starting.
I20260929 18:02:01.796814 26235 tablet_bootstrap.cc:492] T 05609d83481f48cca730afee0f272d9b P ffd8fe4614e5415f9512abb623d1c823: Bootstrap starting.
I20260929 18:02:01.798499 26236 tablet_bootstrap.cc:654] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:01.799085 26235 tablet_bootstrap.cc:654] T 05609d83481f48cca730afee0f272d9b P ffd8fe4614e5415f9512abb623d1c823: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:01.799342 26236 log.cc:824] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:01.800336 26235 log.cc:824] T 05609d83481f48cca730afee0f272d9b P ffd8fe4614e5415f9512abb623d1c823: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:01.801479 26236 tablet_bootstrap.cc:492] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8: No bootstrap required, opened a new log
I20260929 18:02:01.801687 26236 ts_tablet_manager.cc:1397] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:02:01.802441 26235 tablet_bootstrap.cc:492] T 05609d83481f48cca730afee0f272d9b P ffd8fe4614e5415f9512abb623d1c823: No bootstrap required, opened a new log
I20260929 18:02:01.802639 26235 ts_tablet_manager.cc:1397] T 05609d83481f48cca730afee0f272d9b P ffd8fe4614e5415f9512abb623d1c823: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:02:01.802695 26239 tablet_bootstrap.cc:492] T 05609d83481f48cca730afee0f272d9b P ab74e2725dd743c894908611a0085428: Bootstrap starting.
I20260929 18:02:01.805133 26239 tablet_bootstrap.cc:654] T 05609d83481f48cca730afee0f272d9b P ab74e2725dd743c894908611a0085428: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:01.805655 26236 raft_consensus.cc:348] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "146b399202d745fa9e63ef7d05d67fc8" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 41301 } } peers { permanent_uuid: "ab74e2725dd743c894908611a0085428" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37459 } } peers { permanent_uuid: "ffd8fe4614e5415f9512abb623d1c823" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 40057 } }
I20260929 18:02:01.805945 26236 raft_consensus.cc:374] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:01.806000 26236 raft_consensus.cc:729] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 146b399202d745fa9e63ef7d05d67fc8, State: Initialized, Role: FOLLOWER
I20260929 18:02:01.806512 26236 consensus_queue.cc:260] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "146b399202d745fa9e63ef7d05d67fc8" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 41301 } } peers { permanent_uuid: "ab74e2725dd743c894908611a0085428" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37459 } } peers { permanent_uuid: "ffd8fe4614e5415f9512abb623d1c823" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 40057 } }
I20260929 18:02:01.806545 26239 log.cc:824] T 05609d83481f48cca730afee0f272d9b P ab74e2725dd743c894908611a0085428: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:01.806712 26235 raft_consensus.cc:348] T 05609d83481f48cca730afee0f272d9b P ffd8fe4614e5415f9512abb623d1c823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "146b399202d745fa9e63ef7d05d67fc8" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 41301 } } peers { permanent_uuid: "ab74e2725dd743c894908611a0085428" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37459 } } peers { permanent_uuid: "ffd8fe4614e5415f9512abb623d1c823" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 40057 } }
I20260929 18:02:01.807019 26235 raft_consensus.cc:374] T 05609d83481f48cca730afee0f272d9b P ffd8fe4614e5415f9512abb623d1c823 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:01.807087 26235 raft_consensus.cc:729] T 05609d83481f48cca730afee0f272d9b P ffd8fe4614e5415f9512abb623d1c823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd8fe4614e5415f9512abb623d1c823, State: Initialized, Role: FOLLOWER
I20260929 18:02:01.807399 25950 heartbeater.cc:503] Master 127.7.118.62:38479 was elected leader, sending a full tablet report...
I20260929 18:02:01.807588 26235 consensus_queue.cc:260] T 05609d83481f48cca730afee0f272d9b P ffd8fe4614e5415f9512abb623d1c823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "146b399202d745fa9e63ef7d05d67fc8" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 41301 } } peers { permanent_uuid: "ab74e2725dd743c894908611a0085428" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37459 } } peers { permanent_uuid: "ffd8fe4614e5415f9512abb623d1c823" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 40057 } }
I20260929 18:02:01.807682 26236 ts_tablet_manager.cc:1428] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 18:02:01.808521 26235 ts_tablet_manager.cc:1428] T 05609d83481f48cca730afee0f272d9b P ffd8fe4614e5415f9512abb623d1c823: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:02:01.808883 26239 tablet_bootstrap.cc:492] T 05609d83481f48cca730afee0f272d9b P ab74e2725dd743c894908611a0085428: No bootstrap required, opened a new log
I20260929 18:02:01.808893 26084 heartbeater.cc:503] Master 127.7.118.62:38479 was elected leader, sending a full tablet report...
I20260929 18:02:01.809110 26239 ts_tablet_manager.cc:1397] T 05609d83481f48cca730afee0f272d9b P ab74e2725dd743c894908611a0085428: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 18:02:01.813534 26239 raft_consensus.cc:348] T 05609d83481f48cca730afee0f272d9b P ab74e2725dd743c894908611a0085428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "146b399202d745fa9e63ef7d05d67fc8" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 41301 } } peers { permanent_uuid: "ab74e2725dd743c894908611a0085428" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37459 } } peers { permanent_uuid: "ffd8fe4614e5415f9512abb623d1c823" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 40057 } }
I20260929 18:02:01.813887 26239 raft_consensus.cc:374] T 05609d83481f48cca730afee0f272d9b P ab74e2725dd743c894908611a0085428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:01.813931 26239 raft_consensus.cc:729] T 05609d83481f48cca730afee0f272d9b P ab74e2725dd743c894908611a0085428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab74e2725dd743c894908611a0085428, State: Initialized, Role: FOLLOWER
I20260929 18:02:01.814416 26239 consensus_queue.cc:260] T 05609d83481f48cca730afee0f272d9b P ab74e2725dd743c894908611a0085428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "146b399202d745fa9e63ef7d05d67fc8" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 41301 } } peers { permanent_uuid: "ab74e2725dd743c894908611a0085428" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37459 } } peers { permanent_uuid: "ffd8fe4614e5415f9512abb623d1c823" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 40057 } }
I20260929 18:02:01.815110 26218 heartbeater.cc:503] Master 127.7.118.62:38479 was elected leader, sending a full tablet report...
I20260929 18:02:01.815483 26239 ts_tablet_manager.cc:1428] T 05609d83481f48cca730afee0f272d9b P ab74e2725dd743c894908611a0085428: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:02:01.932145 26240 raft_consensus.cc:482] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:02:01.932420 26240 raft_consensus.cc:504] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "146b399202d745fa9e63ef7d05d67fc8" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 41301 } } peers { permanent_uuid: "ab74e2725dd743c894908611a0085428" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37459 } } peers { permanent_uuid: "ffd8fe4614e5415f9512abb623d1c823" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 40057 } }
I20260929 18:02:01.933635 26240 leader_election.cc:288] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab74e2725dd743c894908611a0085428 (127.7.118.3:37459), ffd8fe4614e5415f9512abb623d1c823 (127.7.118.2:40057)
W20260929 18:02:01.935460 25951 tablet.cc:2369] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:02:01.938977 26232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:01.934030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:59273 (local address 127.7.118.3:37459)
0929 18:02:01.934163 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:01.934168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:01.934448 (+ 280us) server_negotiation.cc:407] Connection header received
0929 18:02:01.934604 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:01.934607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:01.934652 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:01.934739 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:02:01.935032 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.935589 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.937482 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.938129 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.938311 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:01.938370 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:01.938744 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 18:02:01.938838 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:02:01.938974 26234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:01.934081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:36803 (local address 127.7.118.2:40057)
0929 18:02:01.934215 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:01.934220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:01.934502 (+ 282us) server_negotiation.cc:407] Connection header received
0929 18:02:01.934629 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:01.934631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:01.934682 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:01.934769 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:02:01.935053 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.935724 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.937482 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.938130 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.938297 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:01.938370 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:01.938744 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 18:02:01.938850 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:02:01.939020 26245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:01.933902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:37459 (local address 127.7.118.1:59273)
0929 18:02:01.934337 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:01.934349 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:01.934485 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:01.934752 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:01.934755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:01.934769 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:02:01.934922 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:01.934927 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.935725 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.935729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:01.937315 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:01.937324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.938331 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.938335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:01.938557 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:01.938603 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 18:02:01.938689 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 18:02:01.938816 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":276,"spinlock_wait_cycles":12160,"thread_start_us":130,"threads_started":1}
I20260929 18:02:01.939026 26244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:01.933915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:40057 (local address 127.7.118.1:36803)
0929 18:02:01.934404 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:01.934416 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:01.934539 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:01.934777 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:01.934779 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:01.934794 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:02:01.934942 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:01.934948 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.935838 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.935841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:01.937311 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:01.937324 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.938331 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.938335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:01.938540 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:01.938600 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 18:02:01.938689 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 18:02:01.938816 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":124,"thread_start_us":146,"threads_started":1}
I20260929 18:02:01.939914 26171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05609d83481f48cca730afee0f272d9b" candidate_uuid: "146b399202d745fa9e63ef7d05d67fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab74e2725dd743c894908611a0085428" is_pre_election: true
I20260929 18:02:01.939973 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05609d83481f48cca730afee0f272d9b" candidate_uuid: "146b399202d745fa9e63ef7d05d67fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffd8fe4614e5415f9512abb623d1c823" is_pre_election: true
I20260929 18:02:01.940279 26171 raft_consensus.cc:2454] T 05609d83481f48cca730afee0f272d9b P ab74e2725dd743c894908611a0085428 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 146b399202d745fa9e63ef7d05d67fc8 in term 0.
I20260929 18:02:01.940296 26038 raft_consensus.cc:2454] T 05609d83481f48cca730afee0f272d9b P ffd8fe4614e5415f9512abb623d1c823 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 146b399202d745fa9e63ef7d05d67fc8 in term 0.
I20260929 18:02:01.940810 25837 leader_election.cc:302] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 146b399202d745fa9e63ef7d05d67fc8, ab74e2725dd743c894908611a0085428; no voters:
I20260929 18:02:01.941138 26240 raft_consensus.cc:2781] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:02:01.941226 26240 raft_consensus.cc:482] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:02:01.941259 26240 raft_consensus.cc:3037] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:01.942509 26240 raft_consensus.cc:504] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "146b399202d745fa9e63ef7d05d67fc8" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 41301 } } peers { permanent_uuid: "ab74e2725dd743c894908611a0085428" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37459 } } peers { permanent_uuid: "ffd8fe4614e5415f9512abb623d1c823" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 40057 } }
I20260929 18:02:01.942942 26240 leader_election.cc:288] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8 [CANDIDATE]: Term 1 election: Requested vote from peers ab74e2725dd743c894908611a0085428 (127.7.118.3:37459), ffd8fe4614e5415f9512abb623d1c823 (127.7.118.2:40057)
I20260929 18:02:01.943357 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05609d83481f48cca730afee0f272d9b" candidate_uuid: "146b399202d745fa9e63ef7d05d67fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffd8fe4614e5415f9512abb623d1c823"
I20260929 18:02:01.943385 26171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05609d83481f48cca730afee0f272d9b" candidate_uuid: "146b399202d745fa9e63ef7d05d67fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab74e2725dd743c894908611a0085428"
I20260929 18:02:01.943519 26038 raft_consensus.cc:3037] T 05609d83481f48cca730afee0f272d9b P ffd8fe4614e5415f9512abb623d1c823 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:01.943533 26171 raft_consensus.cc:3037] T 05609d83481f48cca730afee0f272d9b P ab74e2725dd743c894908611a0085428 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:01.944633 26038 raft_consensus.cc:2454] T 05609d83481f48cca730afee0f272d9b P ffd8fe4614e5415f9512abb623d1c823 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 146b399202d745fa9e63ef7d05d67fc8 in term 1.
I20260929 18:02:01.944789 26171 raft_consensus.cc:2454] T 05609d83481f48cca730afee0f272d9b P ab74e2725dd743c894908611a0085428 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 146b399202d745fa9e63ef7d05d67fc8 in term 1.
I20260929 18:02:01.945176 25840 leader_election.cc:302] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 146b399202d745fa9e63ef7d05d67fc8, ab74e2725dd743c894908611a0085428, ffd8fe4614e5415f9512abb623d1c823; no voters:
I20260929 18:02:01.945384 26240 raft_consensus.cc:2781] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:01.945649 26240 raft_consensus.cc:686] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8 [term 1 LEADER]: Becoming Leader. State: Replica: 146b399202d745fa9e63ef7d05d67fc8, State: Running, Role: LEADER
I20260929 18:02:01.946043 26240 consensus_queue.cc:237] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "146b399202d745fa9e63ef7d05d67fc8" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 41301 } } peers { permanent_uuid: "ab74e2725dd743c894908611a0085428" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37459 } } peers { permanent_uuid: "ffd8fe4614e5415f9512abb623d1c823" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 40057 } }
W20260929 18:02:01.948626 26085 tablet.cc:2369] T 05609d83481f48cca730afee0f272d9b P ffd8fe4614e5415f9512abb623d1c823: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:02:01.949594 25757 catalog_manager.cc:5495] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 146b399202d745fa9e63ef7d05d67fc8 (127.7.118.1). New cstate: current_term: 1 leader_uuid: "146b399202d745fa9e63ef7d05d67fc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "146b399202d745fa9e63ef7d05d67fc8" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab74e2725dd743c894908611a0085428" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffd8fe4614e5415f9512abb623d1c823" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 40057 } health_report { overall_health: UNKNOWN } } }
W20260929 18:02:01.965126 26219 tablet.cc:2369] T 05609d83481f48cca730afee0f272d9b P ab74e2725dd743c894908611a0085428: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:02:01.979374 26231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:01.975513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46256 (local address 127.7.118.1:41301)
0929 18:02:01.975668 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:01.975675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:01.975691 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:02:01.975780 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:01.975783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:01.975838 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:01.975961 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:02:01.976248 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.976801 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.977684 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:01.977945 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:01.978068 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:01.978458 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:02:01.978899 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:02:01.979070 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:01.979170 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 18:02:01.979216 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:02:01.986112 26038 raft_consensus.cc:1261] T 05609d83481f48cca730afee0f272d9b P ffd8fe4614e5415f9512abb623d1c823 [term 1 FOLLOWER]: Refusing update from remote peer 146b399202d745fa9e63ef7d05d67fc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:02:01.986114 26171 raft_consensus.cc:1261] T 05609d83481f48cca730afee0f272d9b P ab74e2725dd743c894908611a0085428 [term 1 FOLLOWER]: Refusing update from remote peer 146b399202d745fa9e63ef7d05d67fc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:02:01.986995 26240 consensus_queue.cc:1035] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffd8fe4614e5415f9512abb623d1c823" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:02:01.987249 26246 consensus_queue.cc:1035] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab74e2725dd743c894908611a0085428" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 37459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:02:01.995725 26249 mvcc.cc:205] Tried to move back new op lower bound from 7334727360446930944 to 7334727360294199296. Current Snapshot: MvccSnapshot[applied={T|T < 7334727360446930944}]
I20260929 18:02:01.997009 26251 mvcc.cc:205] Tried to move back new op lower bound from 7334727360446930944 to 7334727360294199296. Current Snapshot: MvccSnapshot[applied={T|T < 7334727360446930944}]
I20260929 18:02:01.998353 26250 mvcc.cc:205] Tried to move back new op lower bound from 7334727360446930944 to 7334727360294199296. Current Snapshot: MvccSnapshot[applied={T|T < 7334727360446930944}]
I20260929 18:02:02.005571 25758 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:02:02.005784 25758 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:02:02.008415 25758 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5468f7532f41442685f2466829c31a83: Sending DeleteTablet for 3 replicas of tablet 05609d83481f48cca730afee0f272d9b
I20260929 18:02:02.009367 25884 tablet_service.cc:1514] Processing DeleteTablet for tablet 05609d83481f48cca730afee0f272d9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:02:01 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:46242
I20260929 18:02:02.009373 26152 tablet_service.cc:1514] Processing DeleteTablet for tablet 05609d83481f48cca730afee0f272d9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:02:01 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:47284
I20260929 18:02:02.009938 26259 tablet_replica.cc:330] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8: stopping tablet replica
I20260929 18:02:02.009972 26018 tablet_service.cc:1514] Processing DeleteTablet for tablet 05609d83481f48cca730afee0f272d9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:02:01 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:52384
I20260929 18:02:02.010391 26259 raft_consensus.cc:2229] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:02:02.010538 26261 tablet_replica.cc:330] T 05609d83481f48cca730afee0f272d9b P ffd8fe4614e5415f9512abb623d1c823: stopping tablet replica
I20260929 18:02:02.010782 26259 raft_consensus.cc:2258] T 05609d83481f48cca730afee0f272d9b P 146b399202d745fa9e63ef7d05d67fc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:02:02.010985 26261 raft_consensus.cc:2229] T 05609d83481f48cca730afee0f272d9b P ffd8fe4614e5415f9512abb623d1c823 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:02:02.011247 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 25821
I20260929 18:02:02.012007 26260 tablet_replica.cc:330] T 05609d83481f48cca730afee0f272d9b P ab74e2725dd743c894908611a0085428: stopping tablet replica
I20260929 18:02:02.012229 26261 raft_consensus.cc:2258] T 05609d83481f48cca730afee0f272d9b P ffd8fe4614e5415f9512abb623d1c823 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:02:02.012365 26260 raft_consensus.cc:2229] T 05609d83481f48cca730afee0f272d9b P ab74e2725dd743c894908611a0085428 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:02:02.012702 26260 raft_consensus.cc:2258] T 05609d83481f48cca730afee0f272d9b P ab74e2725dd743c894908611a0085428 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:02:02.014315 26261 ts_tablet_manager.cc:1905] T 05609d83481f48cca730afee0f272d9b P ffd8fe4614e5415f9512abb623d1c823: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:02:02.014830 26260 ts_tablet_manager.cc:1905] T 05609d83481f48cca730afee0f272d9b P ab74e2725dd743c894908611a0085428: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:02:02.017331 26261 ts_tablet_manager.cc:1918] T 05609d83481f48cca730afee0f272d9b P ffd8fe4614e5415f9512abb623d1c823: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:02:02.017477 26261 log.cc:1196] T 05609d83481f48cca730afee0f272d9b P ffd8fe4614e5415f9512abb623d1c823: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-1/wal/wals/05609d83481f48cca730afee0f272d9b
I20260929 18:02:02.017560 26260 ts_tablet_manager.cc:1918] T 05609d83481f48cca730afee0f272d9b P ab74e2725dd743c894908611a0085428: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:02:02.017661 26260 log.cc:1196] T 05609d83481f48cca730afee0f272d9b P ab74e2725dd743c894908611a0085428: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704657566865-7640-0/minicluster-data/ts-2/wal/wals/05609d83481f48cca730afee0f272d9b
I20260929 18:02:02.017827 26261 ts_tablet_manager.cc:1939] T 05609d83481f48cca730afee0f272d9b P ffd8fe4614e5415f9512abb623d1c823: Deleting consensus metadata
I20260929 18:02:02.018000 26260 ts_tablet_manager.cc:1939] T 05609d83481f48cca730afee0f272d9b P ab74e2725dd743c894908611a0085428: Deleting consensus metadata
W20260929 18:02:02.019093 25746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.118.1:41301 (error 108)
I20260929 18:02:02.019490 25743 catalog_manager.cc:4841] TS ab74e2725dd743c894908611a0085428 (127.7.118.3:37459): tablet 05609d83481f48cca730afee0f272d9b (table test-table [id=850305b369b54bc6896053000a4c925a]) successfully deleted
I20260929 18:02:02.019703 25746 catalog_manager.cc:4841] TS ffd8fe4614e5415f9512abb623d1c823 (127.7.118.2:40057): tablet 05609d83481f48cca730afee0f272d9b (table test-table [id=850305b369b54bc6896053000a4c925a]) successfully deleted
I20260929 18:02:02.019948 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 25955
I20260929 18:02:02.020432 26229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:02.020049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:41301 (local address 127.0.0.1:46270)
0929 18:02:02.020256 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:02.020331 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.1:41301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260929 18:02:02.024380 25746 catalog_manager.cc:4584] TS 146b399202d745fa9e63ef7d05d67fc8 (127.7.118.1:41301): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 05609d83481f48cca730afee0f272d9b: Network error: Client connection negotiation failed: client connection to 127.7.118.1:41301: connect: Connection refused (error 111)
I20260929 18:02:02.026470 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 26089
I20260929 18:02:02.032524 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 25727
2026-09-29T18:02:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2555 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T18:02:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:02:02Z Disabled control of system clock
I20260929 18:02:02.064911 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:44527
--webserver_interface=127.7.118.62
--webserver_port=0
--builtin_ntp_servers=127.7.118.20:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:44527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:02:02.187175 26266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:02.187496 26266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:02.187534 26266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:02.191910 26266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:02.191990 26266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:02.192005 26266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:02.192024 26266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:02.192040 26266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:02.197147 26266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:44527
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:44527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:02.198357 26266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:02.199798 26266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:02.209944 26272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:02.209875 26274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:02.209903 26271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:02.210538 26266 server_base.cc:1034] running on GCE node
I20260929 18:02:02.211171 26266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:02.212467 26266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:02.213689 26266 hybrid_clock.cc:648] HybridClock initialized: now 1790704922213670 us; error 38 us; skew 500 ppm
I20260929 18:02:02.216116 26266 webserver.cc:460] Webserver started at http://127.7.118.62:40069/ using document root <none> and password file <none>
I20260929 18:02:02.216785 26266 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:02.216886 26266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:02.217104 26266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:02.219472 26266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "b131b252528f4d4c8863fcfde19ad79e"
format_stamp: "Formatted at 2026-09-29 18:02:02 on dist-test-slave-r7ph"
server_key: "2b7b837ade2f3ecc5f8a7e9cc925c976"
server_key_iv: "6a746197f2569474f4a9865c4c2b7477"
server_key_version: "encryptionkey@0"
I20260929 18:02:02.220117 26266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "b131b252528f4d4c8863fcfde19ad79e"
format_stamp: "Formatted at 2026-09-29 18:02:02 on dist-test-slave-r7ph"
server_key: "2b7b837ade2f3ecc5f8a7e9cc925c976"
server_key_iv: "6a746197f2569474f4a9865c4c2b7477"
server_key_version: "encryptionkey@0"
I20260929 18:02:02.224586 26266 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:02:02.228024 26280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:02.229563 26266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 18:02:02.229714 26266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "b131b252528f4d4c8863fcfde19ad79e"
format_stamp: "Formatted at 2026-09-29 18:02:02 on dist-test-slave-r7ph"
server_key: "2b7b837ade2f3ecc5f8a7e9cc925c976"
server_key_iv: "6a746197f2569474f4a9865c4c2b7477"
server_key_version: "encryptionkey@0"
I20260929 18:02:02.229837 26266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:02.244300 26266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:02.245074 26266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:02.245260 26266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:02.252959 26266 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:44527
I20260929 18:02:02.252976 26332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:44527 every 8 connection(s)
I20260929 18:02:02.254025 26266 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:02:02.257557 26333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:02.262336 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 26266
I20260929 18:02:02.262507 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:02:02.262830 7640 external_mini_cluster.cc:1428] Setting key 0151a950f40514e675a054b6e30fe35c
I20260929 18:02:02.264005 26333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b131b252528f4d4c8863fcfde19ad79e: Bootstrap starting.
I20260929 18:02:02.266731 26333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b131b252528f4d4c8863fcfde19ad79e: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:02.267751 26333 log.cc:824] T 00000000000000000000000000000000 P b131b252528f4d4c8863fcfde19ad79e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:02.270421 26333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b131b252528f4d4c8863fcfde19ad79e: No bootstrap required, opened a new log
I20260929 18:02:02.273958 26336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:02.264549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52684 (local address 127.7.118.62:44527)
0929 18:02:02.265306 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:02.265324 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:02.265364 (+ 40us) server_negotiation.cc:407] Connection header received
0929 18:02:02.266129 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:02.266150 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:02.266717 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:02.267096 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:02.267498 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.268573 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.269839 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.270556 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.270756 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:02.271729 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:02.271735 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:02.271751 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:02.272248 (+ 497us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:02.272346 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:02.272371 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:02.272505 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:02.272690 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 18:02:02.273033 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":150,"threads_started":1}
I20260929 18:02:02.274495 26333 raft_consensus.cc:348] T 00000000000000000000000000000000 P b131b252528f4d4c8863fcfde19ad79e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b131b252528f4d4c8863fcfde19ad79e" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 44527 } }
I20260929 18:02:02.274777 26333 raft_consensus.cc:374] T 00000000000000000000000000000000 P b131b252528f4d4c8863fcfde19ad79e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:02.274875 26333 raft_consensus.cc:729] T 00000000000000000000000000000000 P b131b252528f4d4c8863fcfde19ad79e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b131b252528f4d4c8863fcfde19ad79e, State: Initialized, Role: FOLLOWER
I20260929 18:02:02.275424 26333 consensus_queue.cc:260] T 00000000000000000000000000000000 P b131b252528f4d4c8863fcfde19ad79e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b131b252528f4d4c8863fcfde19ad79e" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 44527 } }
I20260929 18:02:02.275597 26333 raft_consensus.cc:388] T 00000000000000000000000000000000 P b131b252528f4d4c8863fcfde19ad79e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:02:02.275655 26333 raft_consensus.cc:482] T 00000000000000000000000000000000 P b131b252528f4d4c8863fcfde19ad79e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:02:02.275745 26333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b131b252528f4d4c8863fcfde19ad79e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:02.277042 26333 raft_consensus.cc:504] T 00000000000000000000000000000000 P b131b252528f4d4c8863fcfde19ad79e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b131b252528f4d4c8863fcfde19ad79e" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 44527 } }
I20260929 18:02:02.277379 26333 leader_election.cc:302] T 00000000000000000000000000000000 P b131b252528f4d4c8863fcfde19ad79e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b131b252528f4d4c8863fcfde19ad79e; no voters:
I20260929 18:02:02.277675 26333 leader_election.cc:288] T 00000000000000000000000000000000 P b131b252528f4d4c8863fcfde19ad79e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:02:02.277913 26338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b131b252528f4d4c8863fcfde19ad79e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:02.278246 26338 raft_consensus.cc:686] T 00000000000000000000000000000000 P b131b252528f4d4c8863fcfde19ad79e [term 1 LEADER]: Becoming Leader. State: Replica: b131b252528f4d4c8863fcfde19ad79e, State: Running, Role: LEADER
I20260929 18:02:02.278636 26338 consensus_queue.cc:237] T 00000000000000000000000000000000 P b131b252528f4d4c8863fcfde19ad79e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b131b252528f4d4c8863fcfde19ad79e" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 44527 } }
I20260929 18:02:02.279096 26333 sys_catalog.cc:564] T 00000000000000000000000000000000 P b131b252528f4d4c8863fcfde19ad79e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:02:02.280442 26340 sys_catalog.cc:455] T 00000000000000000000000000000000 P b131b252528f4d4c8863fcfde19ad79e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b131b252528f4d4c8863fcfde19ad79e. Latest consensus state: current_term: 1 leader_uuid: "b131b252528f4d4c8863fcfde19ad79e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b131b252528f4d4c8863fcfde19ad79e" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 44527 } } }
I20260929 18:02:02.280597 26340 sys_catalog.cc:458] T 00000000000000000000000000000000 P b131b252528f4d4c8863fcfde19ad79e [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:02.281096 26339 sys_catalog.cc:455] T 00000000000000000000000000000000 P b131b252528f4d4c8863fcfde19ad79e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b131b252528f4d4c8863fcfde19ad79e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b131b252528f4d4c8863fcfde19ad79e" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 44527 } } }
I20260929 18:02:02.281241 26339 sys_catalog.cc:458] T 00000000000000000000000000000000 P b131b252528f4d4c8863fcfde19ad79e [sys.catalog]: This master's current role is: LEADER
W20260929 18:02:02.284576 26353 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b131b252528f4d4c8863fcfde19ad79e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:02:02.284727 26353 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:02:02.284822 26348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:02:02.285643 26348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:02:02.292474 26348 catalog_manager.cc:1347] Generated new cluster ID: b95ec91d0e424999bc3ef3f95950f1fd
I20260929 18:02:02.292609 26348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:02:02.310175 26348 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:02:02.311352 26348 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:02:02.324738 26348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b131b252528f4d4c8863fcfde19ad79e: Generated new TSK 0
I20260929 18:02:02.325582 26348 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:02:02.335366 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:0
--local_ip_for_outbound_sockets=127.7.118.1
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:44527
--builtin_ntp_servers=127.7.118.20:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:02:02.456094 26357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:02.456403 26357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:02.456444 26357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:02.460817 26357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:02.460906 26357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:02.460979 26357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 18:02:02.466223 26357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:44527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:02.467518 26357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:02.469275 26357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:02.479800 26363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:02.479790 26362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:02.479790 26365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:02.480121 26357 server_base.cc:1034] running on GCE node
I20260929 18:02:02.480706 26357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:02.481505 26357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:02.482720 26357 hybrid_clock.cc:648] HybridClock initialized: now 1790704922482677 us; error 59 us; skew 500 ppm
I20260929 18:02:02.484920 26357 webserver.cc:460] Webserver started at http://127.7.118.1:40037/ using document root <none> and password file <none>
I20260929 18:02:02.485555 26357 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:02.485656 26357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:02.485910 26357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:02.488140 26357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-0/data/instance:
uuid: "0d681188ba014627812ba853206b038a"
format_stamp: "Formatted at 2026-09-29 18:02:02 on dist-test-slave-r7ph"
server_key: "2bfbb767705d10de468574f8508bb2ce"
server_key_iv: "b5522d6491d083d4537ff520c80f2193"
server_key_version: "encryptionkey@0"
I20260929 18:02:02.488785 26357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance:
uuid: "0d681188ba014627812ba853206b038a"
format_stamp: "Formatted at 2026-09-29 18:02:02 on dist-test-slave-r7ph"
server_key: "2bfbb767705d10de468574f8508bb2ce"
server_key_iv: "b5522d6491d083d4537ff520c80f2193"
server_key_version: "encryptionkey@0"
I20260929 18:02:02.494339 26357 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 18:02:02.497195 26371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:02.498713 26357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:02:02.498852 26357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "0d681188ba014627812ba853206b038a"
format_stamp: "Formatted at 2026-09-29 18:02:02 on dist-test-slave-r7ph"
server_key: "2bfbb767705d10de468574f8508bb2ce"
server_key_iv: "b5522d6491d083d4537ff520c80f2193"
server_key_version: "encryptionkey@0"
I20260929 18:02:02.498986 26357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:02.513484 26357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:02.514359 26357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:02.514571 26357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:02.515206 26357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:02:02.516191 26357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:02:02.516238 26357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:02.516278 26357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:02:02.516332 26357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:02.526041 26357 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:34295
I20260929 18:02:02.526058 26484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:34295 every 8 connection(s)
I20260929 18:02:02.527113 26357 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 18:02:02.532768 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 26357
I20260929 18:02:02.532925 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance
I20260929 18:02:02.533322 7640 external_mini_cluster.cc:1428] Setting key 01d19d4d5a773af46caf5ed27aa198e4
I20260929 18:02:02.535874 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:0
--local_ip_for_outbound_sockets=127.7.118.2
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:44527
--builtin_ntp_servers=127.7.118.20:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:02:02.538460 26336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:02.529088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:50997 (local address 127.7.118.62:44527)
0929 18:02:02.529245 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:02.529250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:02.530855 (+ 1605us) server_negotiation.cc:407] Connection header received
0929 18:02:02.532291 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:02.532295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:02.532362 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:02.532448 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:02.533800 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.534535 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.536024 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.536285 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.536410 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:02.537820 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:02.537824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:02.537826 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:02.537840 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:02.537862 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:02.537871 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:02.537990 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:02.538242 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 18:02:02.538346 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:02:02.539862 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:02.529610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:44527 (local address 127.7.118.1:50997)
0929 18:02:02.530672 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:02.530719 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:02.532038 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:02.532698 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:02.532708 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:02.532824 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:02.533586 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:02.533605 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.534682 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.534685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:02.535878 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:02.535889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.536460 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.536464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:02.536664 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:02.537437 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:02:02.537441 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:02.537511 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:02:02.537619 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:02.537991 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:02.538004 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:02.538012 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 18:02:02.538194 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 18:02:02.538636 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":150,"threads_started":1}
I20260929 18:02:02.541455 26485 heartbeater.cc:347] Connected to a master server at 127.7.118.62:44527
I20260929 18:02:02.541771 26485 heartbeater.cc:464] Registering TS with master...
I20260929 18:02:02.542469 26485 heartbeater.cc:511] Master 127.7.118.62:44527 requested a full tablet report, sending...
I20260929 18:02:02.544226 26297 ts_manager.cc:194] Registered new tserver with Master: 0d681188ba014627812ba853206b038a (127.7.118.1:34295)
I20260929 18:02:02.545554 26297 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.118.1:50997
W20260929 18:02:02.661453 26488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:02.661829 26488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:02.661909 26488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:02.666997 26488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:02.667111 26488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:02.667218 26488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 18:02:02.672675 26488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:44527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:02.673882 26488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:02.675391 26488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:02.685950 26493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:02.685953 26494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:02.685951 26496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:02.686127 26488 server_base.cc:1034] running on GCE node
I20260929 18:02:02.686823 26488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:02.687575 26488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:02.688807 26488 hybrid_clock.cc:648] HybridClock initialized: now 1790704922688792 us; error 39 us; skew 500 ppm
I20260929 18:02:02.690977 26488 webserver.cc:460] Webserver started at http://127.7.118.2:40317/ using document root <none> and password file <none>
I20260929 18:02:02.691609 26488 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:02.691708 26488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:02.691959 26488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:02.694226 26488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-1/data/instance:
uuid: "d5fd69a6ecb64a5494b0ce4b59883747"
format_stamp: "Formatted at 2026-09-29 18:02:02 on dist-test-slave-r7ph"
server_key: "593b334214b8f55b7a2ade25cf9324c1"
server_key_iv: "18d1db129099387b5343a353ccd5a8b0"
server_key_version: "encryptionkey@0"
I20260929 18:02:02.694971 26488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance:
uuid: "d5fd69a6ecb64a5494b0ce4b59883747"
format_stamp: "Formatted at 2026-09-29 18:02:02 on dist-test-slave-r7ph"
server_key: "593b334214b8f55b7a2ade25cf9324c1"
server_key_iv: "18d1db129099387b5343a353ccd5a8b0"
server_key_version: "encryptionkey@0"
I20260929 18:02:02.699323 26488 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 18:02:02.702081 26502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:02.703790 26488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:02:02.703933 26488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "d5fd69a6ecb64a5494b0ce4b59883747"
format_stamp: "Formatted at 2026-09-29 18:02:02 on dist-test-slave-r7ph"
server_key: "593b334214b8f55b7a2ade25cf9324c1"
server_key_iv: "18d1db129099387b5343a353ccd5a8b0"
server_key_version: "encryptionkey@0"
I20260929 18:02:02.704074 26488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:02.717283 26488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:02.718148 26488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:02.718403 26488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:02.719031 26488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:02:02.720038 26488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:02:02.720110 26488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:02.720182 26488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:02:02.720233 26488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:02.729954 26488 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:34979
I20260929 18:02:02.730003 26615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:34979 every 8 connection(s)
I20260929 18:02:02.731050 26488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 18:02:02.734092 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 26488
I20260929 18:02:02.734220 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance
I20260929 18:02:02.734575 7640 external_mini_cluster.cc:1428] Setting key 731119683e92df715000f40fe5b90eeb
I20260929 18:02:02.736737 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:0
--local_ip_for_outbound_sockets=127.7.118.3
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:44527
--builtin_ntp_servers=127.7.118.20:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:02:02.742877 26336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:02.732973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:58969 (local address 127.7.118.62:44527)
0929 18:02:02.733108 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:02.733111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:02.734664 (+ 1553us) server_negotiation.cc:407] Connection header received
0929 18:02:02.736239 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:02.736242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:02.736290 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:02.736369 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:02.737716 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.738392 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.740322 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.740560 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.740652 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:02.742167 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:02.742171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:02.742173 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:02.742223 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:02.742318 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:02.742336 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:02.742426 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:02.742655 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 18:02:02.742731 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:02:02.744323 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:02.733565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:44527 (local address 127.7.118.2:58969)
0929 18:02:02.734466 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:02.734527 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:02.735947 (+ 1420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:02.736607 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:02.736621 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:02.736745 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:02.737527 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:02.737550 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.738548 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.738554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:02.740162 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:02.740178 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.740749 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.740757 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:02.741010 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:02.741755 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:02:02.741759 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:02.741828 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:02:02.741937 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:02.742450 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:02.742463 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:02.742473 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 18:02:02.742645 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 18:02:02.743042 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":125,"threads_started":1}
I20260929 18:02:02.745855 26616 heartbeater.cc:347] Connected to a master server at 127.7.118.62:44527
I20260929 18:02:02.746213 26616 heartbeater.cc:464] Registering TS with master...
I20260929 18:02:02.746868 26616 heartbeater.cc:511] Master 127.7.118.62:44527 requested a full tablet report, sending...
I20260929 18:02:02.748106 26297 ts_manager.cc:194] Registered new tserver with Master: d5fd69a6ecb64a5494b0ce4b59883747 (127.7.118.2:34979)
I20260929 18:02:02.748787 26297 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.118.2:58969
W20260929 18:02:02.855929 26619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:02.856251 26619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:02.856297 26619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:02.860788 26619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:02.860874 26619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:02.860936 26619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 18:02:02.866220 26619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:44527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:02.867477 26619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:02.868929 26619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:02.879777 26625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:02.879886 26619 server_base.cc:1034] running on GCE node
W20260929 18:02:02.879757 26627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:02.879724 26624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:02.880569 26619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:02.881285 26619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:02.882484 26619 hybrid_clock.cc:648] HybridClock initialized: now 1790704922882460 us; error 38 us; skew 500 ppm
I20260929 18:02:02.884558 26619 webserver.cc:460] Webserver started at http://127.7.118.3:44235/ using document root <none> and password file <none>
I20260929 18:02:02.885219 26619 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:02.885325 26619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:02.885573 26619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:02.887797 26619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-2/data/instance:
uuid: "2574299956054998affa0f33f93d6c25"
format_stamp: "Formatted at 2026-09-29 18:02:02 on dist-test-slave-r7ph"
server_key: "434bb8a31b65ec68946ba413773474b8"
server_key_iv: "ffe692e9165ffa2f9cc4c12a48efc280"
server_key_version: "encryptionkey@0"
I20260929 18:02:02.888405 26619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance:
uuid: "2574299956054998affa0f33f93d6c25"
format_stamp: "Formatted at 2026-09-29 18:02:02 on dist-test-slave-r7ph"
server_key: "434bb8a31b65ec68946ba413773474b8"
server_key_iv: "ffe692e9165ffa2f9cc4c12a48efc280"
server_key_version: "encryptionkey@0"
I20260929 18:02:02.892417 26619 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:02:02.895049 26633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:02.896391 26619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:02:02.896530 26619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "2574299956054998affa0f33f93d6c25"
format_stamp: "Formatted at 2026-09-29 18:02:02 on dist-test-slave-r7ph"
server_key: "434bb8a31b65ec68946ba413773474b8"
server_key_iv: "ffe692e9165ffa2f9cc4c12a48efc280"
server_key_version: "encryptionkey@0"
I20260929 18:02:02.896651 26619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:02.919914 26619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:02.920691 26619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:02.920903 26619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:02.921510 26619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:02:02.922520 26619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:02:02.922595 26619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:02.922664 26619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:02:02.922713 26619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:02.931794 26619 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:44855
I20260929 18:02:02.931811 26746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:44855 every 8 connection(s)
I20260929 18:02:02.932761 26619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 18:02:02.934540 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 26619
I20260929 18:02:02.934674 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance
I20260929 18:02:02.934985 7640 external_mini_cluster.cc:1428] Setting key 69619289314fc642be418e395d1e5e92
I20260929 18:02:02.943046 26336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:02.934753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:37363 (local address 127.7.118.62:44527)
0929 18:02:02.934914 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:02.934919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:02.936355 (+ 1436us) server_negotiation.cc:407] Connection header received
0929 18:02:02.938035 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:02.938038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:02.938093 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:02.938170 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:02.939137 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.939675 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.940883 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.941116 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.941229 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:02.942521 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:02.942525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:02.942527 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:02.942541 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:02.942562 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:02.942570 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:02.942656 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:02.942820 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 18:02:02.942893 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 18:02:02.943990 26749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:02.935370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:44527 (local address 127.7.118.3:37363)
0929 18:02:02.936216 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:02.936272 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:02.937822 (+ 1550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:02.938377 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:02.938384 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:02.938472 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:02.938988 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:02.939002 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.939821 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.939824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:02.940741 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:02.940751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.941273 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.941277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:02.941456 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:02.942089 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:02:02.942091 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:02.942137 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:02:02.942341 (+ 204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:02.942663 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:02.942673 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:02.942680 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 18:02:02.942820 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 18:02:02.943144 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":150,"threads_started":1}
I20260929 18:02:02.945283 26747 heartbeater.cc:347] Connected to a master server at 127.7.118.62:44527
I20260929 18:02:02.945604 26747 heartbeater.cc:464] Registering TS with master...
I20260929 18:02:02.946230 26747 heartbeater.cc:511] Master 127.7.118.62:44527 requested a full tablet report, sending...
I20260929 18:02:02.947633 26297 ts_manager.cc:194] Registered new tserver with Master: 2574299956054998affa0f33f93d6c25 (127.7.118.3:44855)
I20260929 18:02:02.948374 26297 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.118.3:37363
I20260929 18:02:02.949951 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:02:02.957062 26336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:02.952559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52688 (local address 127.7.118.62:44527)
0929 18:02:02.952742 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:02.952747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:02.953015 (+ 268us) server_negotiation.cc:407] Connection header received
0929 18:02:02.953204 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:02.953208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:02.953274 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:02.953389 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:02.953695 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.954240 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.955174 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.955411 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.955520 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:02.956522 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:02.956526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:02.956528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:02.956543 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:02.956570 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:02.956579 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:02.956717 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:02.956842 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 18:02:02.956889 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 18:02:02.960728 26297 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:02:02.964082 26297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:02:02.980551 26761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:02.974381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47972 (local address 127.7.118.1:34295)
0929 18:02:02.975403 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:02.975415 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:02.975440 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:02:02.975524 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:02.975530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:02.975765 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:02.975927 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:02:02.976438 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.977195 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.978893 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.979743 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.979926 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:02.980147 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:02.980225 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 18:02:02.980337 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":913,"thread_start_us":118,"threads_started":1}
I20260929 18:02:02.980530 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:02.974173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:34295 (local address 127.0.0.1:47972)
0929 18:02:02.974989 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:02.975004 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:02.975099 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:02.976081 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:02.976084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:02.976103 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:02:02.976284 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:02.976292 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.977325 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.977331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:02.978684 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:02.978696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.979881 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.979884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:02.980011 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:02.980056 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 18:02:02.980171 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 18:02:02.980367 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":4,"thread_start_us":143,"threads_started":1}
I20260929 18:02:02.980792 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:02.974149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:34979 (local address 127.0.0.1:47098)
0929 18:02:02.974899 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:02.974957 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:02.975100 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:02.976063 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:02.976069 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:02.976091 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:02:02.976302 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:02.976309 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.977837 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.977841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:02.979032 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:02.979041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.980436 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.980443 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:02.980532 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:02.980549 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 18:02:02.980594 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 18:02:02.980657 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":86,"thread_start_us":121,"threads_started":1}
I20260929 18:02:02.980906 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:02.975023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52894 (local address 127.7.118.3:44855)
0929 18:02:02.975400 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:02.975415 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:02.975597 (+ 182us) server_negotiation.cc:407] Connection header received
0929 18:02:02.975679 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:02.975684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:02.975881 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:02.976030 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:02:02.976467 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.977705 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.978998 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.979818 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.979925 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:02.980145 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:02.980633 (+ 488us) server_negotiation.cc:299] Negotiation successful
0929 18:02:02.980742 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":93,"threads_started":1}
I20260929 18:02:02.981143 26759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:02.974374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:44855 (local address 127.0.0.1:52894)
0929 18:02:02.975391 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:02.975405 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:02.975504 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:02.976163 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:02.976166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:02.976179 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:02:02.976337 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:02.976356 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.977867 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.977869 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:02.978855 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:02.978864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.980213 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.980216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:02.980367 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:02.980407 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 18:02:02.980472 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 18:02:02.980547 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"mutex_wait_us":63,"thread_start_us":93,"threads_started":1}
I20260929 18:02:02.981561 26763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:02.975041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47098 (local address 127.7.118.2:34979)
0929 18:02:02.975612 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:02.975624 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:02.975644 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:02:02.975705 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:02.975710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:02.975886 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:02.976407 (+ 521us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:02:02.976553 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.977697 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.979428 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:02.980296 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:02.981001 (+ 705us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:02.981233 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:02.981293 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 18:02:02.981399 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":115,"threads_started":1}
I20260929 18:02:02.984071 26419 tablet_service.cc:1467] Processing CreateTablet for tablet 6cb72b28280843c8a35673bd3017f8b4 (DEFAULT_TABLE table=test-table [id=fb126c111624467ba6b7777e275c5661]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:02:02.984761 26550 tablet_service.cc:1467] Processing CreateTablet for tablet 6cb72b28280843c8a35673bd3017f8b4 (DEFAULT_TABLE table=test-table [id=fb126c111624467ba6b7777e275c5661]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:02:02.984764 26681 tablet_service.cc:1467] Processing CreateTablet for tablet 6cb72b28280843c8a35673bd3017f8b4 (DEFAULT_TABLE table=test-table [id=fb126c111624467ba6b7777e275c5661]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:02:02.985301 26419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cb72b28280843c8a35673bd3017f8b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:02.985960 26550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cb72b28280843c8a35673bd3017f8b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:02.985960 26681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cb72b28280843c8a35673bd3017f8b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:02.992376 26764 tablet_bootstrap.cc:492] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a: Bootstrap starting.
I20260929 18:02:02.995333 26766 tablet_bootstrap.cc:492] T 6cb72b28280843c8a35673bd3017f8b4 P 2574299956054998affa0f33f93d6c25: Bootstrap starting.
I20260929 18:02:02.995333 26765 tablet_bootstrap.cc:492] T 6cb72b28280843c8a35673bd3017f8b4 P d5fd69a6ecb64a5494b0ce4b59883747: Bootstrap starting.
I20260929 18:02:02.995496 26764 tablet_bootstrap.cc:654] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:02.996315 26764 log.cc:824] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:02.998107 26765 tablet_bootstrap.cc:654] T 6cb72b28280843c8a35673bd3017f8b4 P d5fd69a6ecb64a5494b0ce4b59883747: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:02.998103 26766 tablet_bootstrap.cc:654] T 6cb72b28280843c8a35673bd3017f8b4 P 2574299956054998affa0f33f93d6c25: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:02.998610 26764 tablet_bootstrap.cc:492] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a: No bootstrap required, opened a new log
I20260929 18:02:02.998867 26764 ts_tablet_manager.cc:1397] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 18:02:02.999234 26766 log.cc:824] T 6cb72b28280843c8a35673bd3017f8b4 P 2574299956054998affa0f33f93d6c25: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:02.999234 26765 log.cc:824] T 6cb72b28280843c8a35673bd3017f8b4 P d5fd69a6ecb64a5494b0ce4b59883747: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:03.002107 26766 tablet_bootstrap.cc:492] T 6cb72b28280843c8a35673bd3017f8b4 P 2574299956054998affa0f33f93d6c25: No bootstrap required, opened a new log
I20260929 18:02:03.002480 26766 ts_tablet_manager.cc:1397] T 6cb72b28280843c8a35673bd3017f8b4 P 2574299956054998affa0f33f93d6c25: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260929 18:02:03.002669 26765 tablet_bootstrap.cc:492] T 6cb72b28280843c8a35673bd3017f8b4 P d5fd69a6ecb64a5494b0ce4b59883747: No bootstrap required, opened a new log
I20260929 18:02:03.002920 26765 ts_tablet_manager.cc:1397] T 6cb72b28280843c8a35673bd3017f8b4 P d5fd69a6ecb64a5494b0ce4b59883747: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260929 18:02:03.003120 26764 raft_consensus.cc:348] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d681188ba014627812ba853206b038a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 34295 } } peers { permanent_uuid: "d5fd69a6ecb64a5494b0ce4b59883747" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34979 } } peers { permanent_uuid: "2574299956054998affa0f33f93d6c25" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 44855 } }
I20260929 18:02:03.003473 26764 raft_consensus.cc:374] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:03.003530 26764 raft_consensus.cc:729] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d681188ba014627812ba853206b038a, State: Initialized, Role: FOLLOWER
I20260929 18:02:03.004078 26764 consensus_queue.cc:260] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d681188ba014627812ba853206b038a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 34295 } } peers { permanent_uuid: "d5fd69a6ecb64a5494b0ce4b59883747" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34979 } } peers { permanent_uuid: "2574299956054998affa0f33f93d6c25" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 44855 } }
I20260929 18:02:03.004900 26485 heartbeater.cc:503] Master 127.7.118.62:44527 was elected leader, sending a full tablet report...
I20260929 18:02:03.005306 26764 ts_tablet_manager.cc:1428] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260929 18:02:03.007238 26765 raft_consensus.cc:348] T 6cb72b28280843c8a35673bd3017f8b4 P d5fd69a6ecb64a5494b0ce4b59883747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d681188ba014627812ba853206b038a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 34295 } } peers { permanent_uuid: "d5fd69a6ecb64a5494b0ce4b59883747" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34979 } } peers { permanent_uuid: "2574299956054998affa0f33f93d6c25" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 44855 } }
I20260929 18:02:03.007247 26766 raft_consensus.cc:348] T 6cb72b28280843c8a35673bd3017f8b4 P 2574299956054998affa0f33f93d6c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d681188ba014627812ba853206b038a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 34295 } } peers { permanent_uuid: "d5fd69a6ecb64a5494b0ce4b59883747" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34979 } } peers { permanent_uuid: "2574299956054998affa0f33f93d6c25" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 44855 } }
I20260929 18:02:03.007575 26765 raft_consensus.cc:374] T 6cb72b28280843c8a35673bd3017f8b4 P d5fd69a6ecb64a5494b0ce4b59883747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:03.007586 26766 raft_consensus.cc:374] T 6cb72b28280843c8a35673bd3017f8b4 P 2574299956054998affa0f33f93d6c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:03.007642 26766 raft_consensus.cc:729] T 6cb72b28280843c8a35673bd3017f8b4 P 2574299956054998affa0f33f93d6c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2574299956054998affa0f33f93d6c25, State: Initialized, Role: FOLLOWER
I20260929 18:02:03.007642 26765 raft_consensus.cc:729] T 6cb72b28280843c8a35673bd3017f8b4 P d5fd69a6ecb64a5494b0ce4b59883747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5fd69a6ecb64a5494b0ce4b59883747, State: Initialized, Role: FOLLOWER
I20260929 18:02:03.008219 26766 consensus_queue.cc:260] T 6cb72b28280843c8a35673bd3017f8b4 P 2574299956054998affa0f33f93d6c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d681188ba014627812ba853206b038a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 34295 } } peers { permanent_uuid: "d5fd69a6ecb64a5494b0ce4b59883747" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34979 } } peers { permanent_uuid: "2574299956054998affa0f33f93d6c25" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 44855 } }
I20260929 18:02:03.008213 26765 consensus_queue.cc:260] T 6cb72b28280843c8a35673bd3017f8b4 P d5fd69a6ecb64a5494b0ce4b59883747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d681188ba014627812ba853206b038a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 34295 } } peers { permanent_uuid: "d5fd69a6ecb64a5494b0ce4b59883747" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34979 } } peers { permanent_uuid: "2574299956054998affa0f33f93d6c25" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 44855 } }
I20260929 18:02:03.009353 26616 heartbeater.cc:503] Master 127.7.118.62:44527 was elected leader, sending a full tablet report...
I20260929 18:02:03.009428 26765 ts_tablet_manager.cc:1428] T 6cb72b28280843c8a35673bd3017f8b4 P d5fd69a6ecb64a5494b0ce4b59883747: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260929 18:02:03.009428 26766 ts_tablet_manager.cc:1428] T 6cb72b28280843c8a35673bd3017f8b4 P 2574299956054998affa0f33f93d6c25: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 18:02:03.009815 26747 heartbeater.cc:503] Master 127.7.118.62:44527 was elected leader, sending a full tablet report...
W20260929 18:02:03.028448 26486 tablet.cc:2369] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:02:03.083415 26770 raft_consensus.cc:482] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:02:03.083679 26770 raft_consensus.cc:504] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d681188ba014627812ba853206b038a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 34295 } } peers { permanent_uuid: "d5fd69a6ecb64a5494b0ce4b59883747" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34979 } } peers { permanent_uuid: "2574299956054998affa0f33f93d6c25" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 44855 } }
I20260929 18:02:03.085016 26770 leader_election.cc:288] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5fd69a6ecb64a5494b0ce4b59883747 (127.7.118.2:34979), 2574299956054998affa0f33f93d6c25 (127.7.118.3:44855)
I20260929 18:02:03.090795 26763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:03.085498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:37805 (local address 127.7.118.2:34979)
0929 18:02:03.085736 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:03.085741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:03.085983 (+ 242us) server_negotiation.cc:407] Connection header received
0929 18:02:03.086047 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:03.086050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:03.086106 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:03.086203 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:02:03.086691 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:03.087519 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:03.089344 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:03.089984 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:03.090105 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:03.090195 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:03.090556 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 18:02:03.090649 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260929 18:02:03.090795 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:03.085698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:56563 (local address 127.7.118.3:44855)
0929 18:02:03.085821 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:03.085824 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:03.085838 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:02:03.086052 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:03.086055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:03.086106 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:03.086204 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:02:03.086691 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:03.087478 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:03.089347 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:03.089984 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:03.090105 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:03.090195 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:03.090556 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 18:02:03.090649 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:02:03.090817 26774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:03.085336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:34979 (local address 127.7.118.1:37805)
0929 18:02:03.085868 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:03.085880 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:03.085971 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:03.086235 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:03.086238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:03.086297 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:02:03.086545 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:03.086552 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:03.087663 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:03.087665 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:03.089175 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:03.089188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:03.090161 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:03.090165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:03.090365 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:03.090416 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 18:02:03.090504 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 18:02:03.090612 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":358,"spinlock_wait_cycles":3712,"thread_start_us":136,"threads_started":1}
I20260929 18:02:03.090817 26773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:03.085336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:44855 (local address 127.7.118.1:56563)
0929 18:02:03.085723 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:03.085741 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:03.085860 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:03.086235 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:03.086238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:03.086292 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:02:03.086544 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:03.086552 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:03.087661 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:03.087666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:03.089174 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:03.089188 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:03.090188 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:03.090191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:03.090370 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:03.090420 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 18:02:03.090504 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 18:02:03.090612 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":139,"spinlock_wait_cycles":896,"thread_start_us":108,"threads_started":1}
I20260929 18:02:03.091686 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb72b28280843c8a35673bd3017f8b4" candidate_uuid: "0d681188ba014627812ba853206b038a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5fd69a6ecb64a5494b0ce4b59883747" is_pre_election: true
I20260929 18:02:03.092074 26570 raft_consensus.cc:2454] T 6cb72b28280843c8a35673bd3017f8b4 P d5fd69a6ecb64a5494b0ce4b59883747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d681188ba014627812ba853206b038a in term 0.
I20260929 18:02:03.092124 26701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb72b28280843c8a35673bd3017f8b4" candidate_uuid: "0d681188ba014627812ba853206b038a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2574299956054998affa0f33f93d6c25" is_pre_election: true
I20260929 18:02:03.092450 26701 raft_consensus.cc:2454] T 6cb72b28280843c8a35673bd3017f8b4 P 2574299956054998affa0f33f93d6c25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d681188ba014627812ba853206b038a in term 0.
I20260929 18:02:03.092703 26373 leader_election.cc:302] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d681188ba014627812ba853206b038a, d5fd69a6ecb64a5494b0ce4b59883747; no voters:
I20260929 18:02:03.093009 26770 raft_consensus.cc:2781] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:02:03.093099 26770 raft_consensus.cc:482] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:02:03.093142 26770 raft_consensus.cc:3037] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:03.094332 26770 raft_consensus.cc:504] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d681188ba014627812ba853206b038a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 34295 } } peers { permanent_uuid: "d5fd69a6ecb64a5494b0ce4b59883747" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34979 } } peers { permanent_uuid: "2574299956054998affa0f33f93d6c25" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 44855 } }
I20260929 18:02:03.094713 26770 leader_election.cc:288] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a [CANDIDATE]: Term 1 election: Requested vote from peers d5fd69a6ecb64a5494b0ce4b59883747 (127.7.118.2:34979), 2574299956054998affa0f33f93d6c25 (127.7.118.3:44855)
I20260929 18:02:03.095074 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb72b28280843c8a35673bd3017f8b4" candidate_uuid: "0d681188ba014627812ba853206b038a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5fd69a6ecb64a5494b0ce4b59883747"
I20260929 18:02:03.095173 26701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb72b28280843c8a35673bd3017f8b4" candidate_uuid: "0d681188ba014627812ba853206b038a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2574299956054998affa0f33f93d6c25"
I20260929 18:02:03.095233 26570 raft_consensus.cc:3037] T 6cb72b28280843c8a35673bd3017f8b4 P d5fd69a6ecb64a5494b0ce4b59883747 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:03.095301 26701 raft_consensus.cc:3037] T 6cb72b28280843c8a35673bd3017f8b4 P 2574299956054998affa0f33f93d6c25 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:03.096668 26701 raft_consensus.cc:2454] T 6cb72b28280843c8a35673bd3017f8b4 P 2574299956054998affa0f33f93d6c25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d681188ba014627812ba853206b038a in term 1.
I20260929 18:02:03.096663 26570 raft_consensus.cc:2454] T 6cb72b28280843c8a35673bd3017f8b4 P d5fd69a6ecb64a5494b0ce4b59883747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d681188ba014627812ba853206b038a in term 1.
I20260929 18:02:03.097041 26373 leader_election.cc:302] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d681188ba014627812ba853206b038a, d5fd69a6ecb64a5494b0ce4b59883747; no voters:
I20260929 18:02:03.097278 26770 raft_consensus.cc:2781] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:03.097530 26770 raft_consensus.cc:686] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a [term 1 LEADER]: Becoming Leader. State: Replica: 0d681188ba014627812ba853206b038a, State: Running, Role: LEADER
I20260929 18:02:03.097939 26770 consensus_queue.cc:237] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d681188ba014627812ba853206b038a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 34295 } } peers { permanent_uuid: "d5fd69a6ecb64a5494b0ce4b59883747" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34979 } } peers { permanent_uuid: "2574299956054998affa0f33f93d6c25" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 44855 } }
I20260929 18:02:03.102124 26292 catalog_manager.cc:5495] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d681188ba014627812ba853206b038a (127.7.118.1). New cstate: current_term: 1 leader_uuid: "0d681188ba014627812ba853206b038a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d681188ba014627812ba853206b038a" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 34295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5fd69a6ecb64a5494b0ce4b59883747" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2574299956054998affa0f33f93d6c25" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 44855 } health_report { overall_health: UNKNOWN } } }
I20260929 18:02:03.141750 26761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:03.137425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47974 (local address 127.7.118.1:34295)
0929 18:02:03.137566 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:03.137570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:03.137585 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:02:03.137663 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:03.137665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:03.137726 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:03.137837 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:02:03.138181 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:03.138761 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:03.139923 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:03.140198 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:03.140291 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:03.140747 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:02:03.141221 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:02:03.141371 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:03.141557 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 18:02:03.141601 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:02:03.148523 26570 raft_consensus.cc:1261] T 6cb72b28280843c8a35673bd3017f8b4 P d5fd69a6ecb64a5494b0ce4b59883747 [term 1 FOLLOWER]: Refusing update from remote peer 0d681188ba014627812ba853206b038a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:02:03.148523 26701 raft_consensus.cc:1261] T 6cb72b28280843c8a35673bd3017f8b4 P 2574299956054998affa0f33f93d6c25 [term 1 FOLLOWER]: Refusing update from remote peer 0d681188ba014627812ba853206b038a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:02:03.149528 26770 consensus_queue.cc:1035] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2574299956054998affa0f33f93d6c25" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 44855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:02:03.149874 26775 consensus_queue.cc:1035] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5fd69a6ecb64a5494b0ce4b59883747" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:02:03.157486 26778 mvcc.cc:205] Tried to move back new op lower bound from 7334727365206896640 to 7334727365012807680. Current Snapshot: MvccSnapshot[applied={T|T < 7334727365206896640}]
I20260929 18:02:03.167312 26297 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:02:03.167500 26297 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:02:03.170650 26297 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b131b252528f4d4c8863fcfde19ad79e: Sending DeleteTablet for 3 replicas of tablet 6cb72b28280843c8a35673bd3017f8b4
I20260929 18:02:03.171782 26681 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cb72b28280843c8a35673bd3017f8b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:02:03 UTC) from {username='slave'} at 127.0.0.1:52894
I20260929 18:02:03.171813 26419 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cb72b28280843c8a35673bd3017f8b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:02:03 UTC) from {username='slave'} at 127.0.0.1:47972
I20260929 18:02:03.171783 26550 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cb72b28280843c8a35673bd3017f8b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:02:03 UTC) from {username='slave'} at 127.0.0.1:47098
I20260929 18:02:03.172467 26789 tablet_replica.cc:330] T 6cb72b28280843c8a35673bd3017f8b4 P 2574299956054998affa0f33f93d6c25: stopping tablet replica
I20260929 18:02:03.172745 26787 tablet_replica.cc:330] T 6cb72b28280843c8a35673bd3017f8b4 P d5fd69a6ecb64a5494b0ce4b59883747: stopping tablet replica
I20260929 18:02:03.172889 26789 raft_consensus.cc:2229] T 6cb72b28280843c8a35673bd3017f8b4 P 2574299956054998affa0f33f93d6c25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:02:03.172955 26788 tablet_replica.cc:330] T 6cb72b28280843c8a35673bd3017f8b4 P 0d681188ba014627812ba853206b038a: stopping tablet replica
I20260929 18:02:03.173252 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 26357
I20260929 18:02:03.173537 26789 raft_consensus.cc:2258] T 6cb72b28280843c8a35673bd3017f8b4 P 2574299956054998affa0f33f93d6c25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:02:03.174146 26787 raft_consensus.cc:2229] T 6cb72b28280843c8a35673bd3017f8b4 P d5fd69a6ecb64a5494b0ce4b59883747 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:02:03.174832 26787 raft_consensus.cc:2258] T 6cb72b28280843c8a35673bd3017f8b4 P d5fd69a6ecb64a5494b0ce4b59883747 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:02:03.176142 26789 ts_tablet_manager.cc:1905] T 6cb72b28280843c8a35673bd3017f8b4 P 2574299956054998affa0f33f93d6c25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:02:03.177925 26787 ts_tablet_manager.cc:1905] T 6cb72b28280843c8a35673bd3017f8b4 P d5fd69a6ecb64a5494b0ce4b59883747: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:02:03.179430 26789 ts_tablet_manager.cc:1918] T 6cb72b28280843c8a35673bd3017f8b4 P 2574299956054998affa0f33f93d6c25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:02:03.179553 26789 log.cc:1196] T 6cb72b28280843c8a35673bd3017f8b4 P 2574299956054998affa0f33f93d6c25: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704657566865-7640-0/minicluster-data/ts-2/wal/wals/6cb72b28280843c8a35673bd3017f8b4
I20260929 18:02:03.179939 26789 ts_tablet_manager.cc:1939] T 6cb72b28280843c8a35673bd3017f8b4 P 2574299956054998affa0f33f93d6c25: Deleting consensus metadata
I20260929 18:02:03.180531 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 26488
W20260929 18:02:03.181392 26281 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.118.1:34295 (error 108)
I20260929 18:02:03.181483 26283 catalog_manager.cc:4841] TS 2574299956054998affa0f33f93d6c25 (127.7.118.3:44855): tablet 6cb72b28280843c8a35673bd3017f8b4 (table test-table [id=fb126c111624467ba6b7777e275c5661]) successfully deleted
I20260929 18:02:03.182842 26759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:03.182460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:34295 (local address 127.0.0.1:47988)
0929 18:02:03.182676 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:03.182750 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.1:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260929 18:02:03.183014 26281 catalog_manager.cc:4584] TS 0d681188ba014627812ba853206b038a (127.7.118.1:34295): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6cb72b28280843c8a35673bd3017f8b4: Network error: Client connection negotiation failed: client connection to 127.7.118.1:34295: connect: Connection refused (error 111)
I20260929 18:02:03.187356 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 26619
I20260929 18:02:03.188246 26759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:03.187603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:34979 (local address 127.0.0.1:47100)
0929 18:02:03.188052 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:03.188125 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.2:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343}
I20260929 18:02:03.193858 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 26266
2026-09-29T18:02:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1161 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:02:03 dist-test-slave-r7ph krb5kdc[26793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:02:03 dist-test-slave-r7ph krb5kdc[26793](info): set up 2 sockets
Sep 29 18:02:03 dist-test-slave-r7ph krb5kdc[26793](info): commencing operation
krb5kdc: starting...
W20260929 18:02:04.266531 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 18:02:04 dist-test-slave-r7ph krb5kdc[26793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:02:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:02:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 18:02:04.403228 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:33971
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:33971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:04.524878 26806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:04.525223 26806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:04.525287 26806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:04.530164 26806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:04.530310 26806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:04.530364 26806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:04.530400 26806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:04.530439 26806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:04.536666 26806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:33971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:33971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:04.537875 26806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:04.539059 26806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:04.548695 26812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:04.548728 26811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:04.548856 26806 server_base.cc:1034] running on GCE node
W20260929 18:02:04.549082 26814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:04.549808 26806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:04.551117 26806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:04.552359 26806 hybrid_clock.cc:648] HybridClock initialized: now 1790704924552343 us; error 35 us; skew 500 ppm
Sep 29 18:02:04 dist-test-slave-r7ph krb5kdc[26793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704924, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:04.555955 26806 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 18:02:04.557001 26806 webserver.cc:460] Webserver started at http://127.7.118.62:43739/ using document root <none> and password file <none>
I20260929 18:02:04.557641 26806 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:04.557708 26806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:04.557940 26806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:04.560249 26806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "2b6b023f9c1a42b29a3a5d75e18c3848"
format_stamp: "Formatted at 2026-09-29 18:02:04 on dist-test-slave-r7ph"
server_key: "a020c77500680052d65bdc56a5e9b242"
server_key_iv: "e6ca1803288ed2e2b5f92ad97e44feaf"
server_key_version: "encryptionkey@0"
I20260929 18:02:04.560822 26806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "2b6b023f9c1a42b29a3a5d75e18c3848"
format_stamp: "Formatted at 2026-09-29 18:02:04 on dist-test-slave-r7ph"
server_key: "a020c77500680052d65bdc56a5e9b242"
server_key_iv: "e6ca1803288ed2e2b5f92ad97e44feaf"
server_key_version: "encryptionkey@0"
I20260929 18:02:04.564910 26806 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 18:02:04.567749 26821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:04.569190 26806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 18:02:04.569386 26806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "2b6b023f9c1a42b29a3a5d75e18c3848"
format_stamp: "Formatted at 2026-09-29 18:02:04 on dist-test-slave-r7ph"
server_key: "a020c77500680052d65bdc56a5e9b242"
server_key_iv: "e6ca1803288ed2e2b5f92ad97e44feaf"
server_key_version: "encryptionkey@0"
I20260929 18:02:04.569543 26806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:04.586498 26806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:04.590430 26806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:04.590610 26806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:04.598165 26806 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:33971
I20260929 18:02:04.598208 26873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:33971 every 8 connection(s)
I20260929 18:02:04.599274 26806 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:02:04.600504 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 26806
I20260929 18:02:04.600682 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:02:04.600950 7640 external_mini_cluster.cc:1428] Setting key 8a0aed5f2a422a78fc71f67c8fc39868
I20260929 18:02:04.603226 26874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:04.609162 26874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b6b023f9c1a42b29a3a5d75e18c3848: Bootstrap starting.
Sep 29 18:02:04 dist-test-slave-r7ph krb5kdc[26793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:02:04.611963 26874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b6b023f9c1a42b29a3a5d75e18c3848: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:04.612773 26874 log.cc:824] T 00000000000000000000000000000000 P 2b6b023f9c1a42b29a3a5d75e18c3848: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:04.615566 26874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b6b023f9c1a42b29a3a5d75e18c3848: No bootstrap required, opened a new log
I20260929 18:02:04.616919 26877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:04.602403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42548 (local address 127.7.118.62:33971)
0929 18:02:04.602974 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:04.602990 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:04.603029 (+ 39us) server_negotiation.cc:407] Connection header received
0929 18:02:04.603814 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:04.603834 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:04.604197 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:04.604564 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:04.605426 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:04.606573 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:04.607468 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:04.607974 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:04.608186 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:04.611275 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:04.611281 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:04.611296 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:04.611331 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:04.613417 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:04.614175 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:04.614181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:04.614186 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:04.614373 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:04.614706 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:04.614709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:04.614710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:04.615169 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:04.615334 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:04.615728 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 18:02:04.616100 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":160,"threads_started":1}
I20260929 18:02:04.619855 26874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b6b023f9c1a42b29a3a5d75e18c3848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6b023f9c1a42b29a3a5d75e18c3848" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33971 } }
I20260929 18:02:04.620115 26874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b6b023f9c1a42b29a3a5d75e18c3848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:04.620187 26874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b6b023f9c1a42b29a3a5d75e18c3848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b6b023f9c1a42b29a3a5d75e18c3848, State: Initialized, Role: FOLLOWER
I20260929 18:02:04.620674 26874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b6b023f9c1a42b29a3a5d75e18c3848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6b023f9c1a42b29a3a5d75e18c3848" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33971 } }
I20260929 18:02:04.620805 26874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b6b023f9c1a42b29a3a5d75e18c3848 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:02:04.620891 26874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b6b023f9c1a42b29a3a5d75e18c3848 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:02:04.620992 26874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b6b023f9c1a42b29a3a5d75e18c3848 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:04.622128 26874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b6b023f9c1a42b29a3a5d75e18c3848 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6b023f9c1a42b29a3a5d75e18c3848" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33971 } }
I20260929 18:02:04.622588 26874 leader_election.cc:302] T 00000000000000000000000000000000 P 2b6b023f9c1a42b29a3a5d75e18c3848 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b6b023f9c1a42b29a3a5d75e18c3848; no voters:
I20260929 18:02:04.622992 26874 leader_election.cc:288] T 00000000000000000000000000000000 P 2b6b023f9c1a42b29a3a5d75e18c3848 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:02:04.623121 26879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b6b023f9c1a42b29a3a5d75e18c3848 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:04.623306 26879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b6b023f9c1a42b29a3a5d75e18c3848 [term 1 LEADER]: Becoming Leader. State: Replica: 2b6b023f9c1a42b29a3a5d75e18c3848, State: Running, Role: LEADER
I20260929 18:02:04.623628 26879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b6b023f9c1a42b29a3a5d75e18c3848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6b023f9c1a42b29a3a5d75e18c3848" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33971 } }
I20260929 18:02:04.624405 26874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b6b023f9c1a42b29a3a5d75e18c3848 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:02:04.625500 26880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b6b023f9c1a42b29a3a5d75e18c3848 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b6b023f9c1a42b29a3a5d75e18c3848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6b023f9c1a42b29a3a5d75e18c3848" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33971 } } }
I20260929 18:02:04.625648 26880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b6b023f9c1a42b29a3a5d75e18c3848 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:04.626034 26881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b6b023f9c1a42b29a3a5d75e18c3848 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b6b023f9c1a42b29a3a5d75e18c3848. Latest consensus state: current_term: 1 leader_uuid: "2b6b023f9c1a42b29a3a5d75e18c3848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6b023f9c1a42b29a3a5d75e18c3848" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33971 } } }
I20260929 18:02:04.626140 26881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b6b023f9c1a42b29a3a5d75e18c3848 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:04.626590 26888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:02:04.629441 26888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:02:04.635934 26888 catalog_manager.cc:1347] Generated new cluster ID: 1ac06a8d95934661b041b3371e070e88
I20260929 18:02:04.636034 26888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:02:04.654739 26888 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:02:04.656389 26888 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:02:04.668965 26888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b6b023f9c1a42b29a3a5d75e18c3848: Generated new TSK 0
I20260929 18:02:04.670002 26888 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.118.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
Entry for principal kudu/127.7.118.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab.
I20260929 18:02:04.703192 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:0
--local_ip_for_outbound_sockets=127.7.118.1
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:33971
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:04.830602 26900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:04.830941 26900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:04.831022 26900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:04.835395 26900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:04.835505 26900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:04.835605 26900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 18:02:04.841236 26900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.1.keytab
--principal=kudu/127.7.118.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:33971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:04.842555 26900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:04.843622 26900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:04.854305 26905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:04.854332 26906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:04.854422 26900 server_base.cc:1034] running on GCE node
W20260929 18:02:04.854372 26908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:04.855027 26900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:04.855768 26900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:04.856958 26900 hybrid_clock.cc:648] HybridClock initialized: now 1790704924856873 us; error 99 us; skew 500 ppm
Sep 29 18:02:04 dist-test-slave-r7ph krb5kdc[26793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704924, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:04.860482 26900 init.cc:377] Logged in from keytab as kudu/127.7.118.1@KRBTEST.COM (short username kudu)
I20260929 18:02:04.861922 26900 webserver.cc:460] Webserver started at http://127.7.118.1:37611/ using document root <none> and password file <none>
I20260929 18:02:04.862671 26900 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:04.862780 26900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:04.863045 26900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:04.865352 26900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-0/data/instance:
uuid: "c2c613a0dedb4402b485e679fe5f6a0c"
format_stamp: "Formatted at 2026-09-29 18:02:04 on dist-test-slave-r7ph"
server_key: "b2558036e1040f6cb2bb51be14605f18"
server_key_iv: "5bb44dd542c546d8b546a799a1f63b0f"
server_key_version: "encryptionkey@0"
I20260929 18:02:04.865999 26900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance:
uuid: "c2c613a0dedb4402b485e679fe5f6a0c"
format_stamp: "Formatted at 2026-09-29 18:02:04 on dist-test-slave-r7ph"
server_key: "b2558036e1040f6cb2bb51be14605f18"
server_key_iv: "5bb44dd542c546d8b546a799a1f63b0f"
server_key_version: "encryptionkey@0"
I20260929 18:02:04.870330 26900 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:02:04.873060 26915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:04.874758 26900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:02:04.874912 26900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "c2c613a0dedb4402b485e679fe5f6a0c"
format_stamp: "Formatted at 2026-09-29 18:02:04 on dist-test-slave-r7ph"
server_key: "b2558036e1040f6cb2bb51be14605f18"
server_key_iv: "5bb44dd542c546d8b546a799a1f63b0f"
server_key_version: "encryptionkey@0"
I20260929 18:02:04.875025 26900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:04.899677 26900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:04.903930 26900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:04.904126 26900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:04.904750 26900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:02:04.905611 26900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:02:04.905663 26900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:04.905706 26900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:02:04.905722 26900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:04.916272 26900 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:39939
I20260929 18:02:04.916288 27028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:39939 every 8 connection(s)
I20260929 18:02:04.917315 26900 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 18:02:04.921909 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 26900
I20260929 18:02:04.922087 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance
I20260929 18:02:04.922498 7640 external_mini_cluster.cc:1428] Setting key 987faa1ccb2e254698917b943e4a7532
Sep 29 18:02:04 dist-test-slave-r7ph krb5kdc[26793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704924, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.1@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.118.2@KRBTEST.COM; defaulting to no policy
I20260929 18:02:04.935545 26877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:04.919631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:37047 (local address 127.7.118.62:33971)
0929 18:02:04.919823 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:04.919827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:04.920908 (+ 1081us) server_negotiation.cc:407] Connection header received
0929 18:02:04.921964 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:04.921968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:04.922036 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:04.922133 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:04.923943 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:04.924800 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:04.926694 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:04.927135 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:04.927275 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:04.930589 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:04.930596 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:04.930599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:04.930636 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:04.932690 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:04.933475 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:04.933479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:04.933480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:04.933567 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:04.934069 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:04.934073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:04.934074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:04.934417 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:04.934508 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:04.935262 (+ 754us) server_negotiation.cc:299] Negotiation successful
0929 18:02:04.935411 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 18:02:04.936626 27031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:04.919904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33971 (local address 127.7.118.1:37047)
0929 18:02:04.920723 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:04.920779 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:04.921677 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:04.922436 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:04.922448 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:04.923039 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:04.923727 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:04.923748 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:04.925006 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:04.925013 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:04.926554 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:04.926565 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:04.927347 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:04.927350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:04.927533 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:04.928366 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:02:04.928387 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:04.930349 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:04.932868 (+ 2519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:04.932874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:04.932902 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:04.933332 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:04.933694 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:04.933698 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:04.933699 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:04.933901 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:04.934572 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:04.934579 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:04.934936 (+ 357us) client_negotiation.cc:769] Sending connection context
0929 18:02:04.935149 (+ 213us) client_negotiation.cc:240] Negotiation successful
0929 18:02:04.935479 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":130,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.2@KRBTEST.COM" created.
I20260929 18:02:04.938361 27029 heartbeater.cc:347] Connected to a master server at 127.7.118.62:33971
I20260929 18:02:04.938688 27029 heartbeater.cc:464] Registering TS with master...
I20260929 18:02:04.939564 27029 heartbeater.cc:511] Master 127.7.118.62:33971 requested a full tablet report, sending...
I20260929 18:02:04.941717 26838 ts_manager.cc:194] Registered new tserver with Master: c2c613a0dedb4402b485e679fe5f6a0c (127.7.118.1:39939)
I20260929 18:02:04.943087 26838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.1@KRBTEST.COM'} at 127.7.118.1:37047
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
Entry for principal kudu/127.7.118.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab.
I20260929 18:02:04.959966 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:0
--local_ip_for_outbound_sockets=127.7.118.2
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:33971
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:05.081395 27034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:05.081717 27034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:05.081763 27034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:05.086074 27034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:05.086221 27034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:05.086357 27034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 18:02:05.092172 27034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.2.keytab
--principal=kudu/127.7.118.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:33971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:05.093508 27034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:05.094656 27034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:05.105809 27040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:05.105837 27039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:05.105811 27042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:05.105885 27034 server_base.cc:1034] running on GCE node
I20260929 18:02:05.106722 27034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:05.107534 27034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:05.108757 27034 hybrid_clock.cc:648] HybridClock initialized: now 1790704925108729 us; error 49 us; skew 500 ppm
Sep 29 18:02:05 dist-test-slave-r7ph krb5kdc[26793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704925, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:05.112202 27034 init.cc:377] Logged in from keytab as kudu/127.7.118.2@KRBTEST.COM (short username kudu)
I20260929 18:02:05.113260 27034 webserver.cc:460] Webserver started at http://127.7.118.2:33705/ using document root <none> and password file <none>
I20260929 18:02:05.113978 27034 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:05.114086 27034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:05.114403 27034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:05.116752 27034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-1/data/instance:
uuid: "ac5ae078ef7842cfb1cc7d0ce63c83a3"
format_stamp: "Formatted at 2026-09-29 18:02:05 on dist-test-slave-r7ph"
server_key: "7573f3f9399ffffdb556777be50918ca"
server_key_iv: "1d19456fd358a7aefa6a0e25c58f88f1"
server_key_version: "encryptionkey@0"
I20260929 18:02:05.117427 27034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance:
uuid: "ac5ae078ef7842cfb1cc7d0ce63c83a3"
format_stamp: "Formatted at 2026-09-29 18:02:05 on dist-test-slave-r7ph"
server_key: "7573f3f9399ffffdb556777be50918ca"
server_key_iv: "1d19456fd358a7aefa6a0e25c58f88f1"
server_key_version: "encryptionkey@0"
I20260929 18:02:05.122473 27034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 18:02:05.125669 27049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:05.127393 27034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:02:05.127522 27034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "ac5ae078ef7842cfb1cc7d0ce63c83a3"
format_stamp: "Formatted at 2026-09-29 18:02:05 on dist-test-slave-r7ph"
server_key: "7573f3f9399ffffdb556777be50918ca"
server_key_iv: "1d19456fd358a7aefa6a0e25c58f88f1"
server_key_version: "encryptionkey@0"
I20260929 18:02:05.127635 27034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:05.141541 27034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:05.145031 27034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:05.145290 27034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:05.145967 27034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:02:05.146938 27034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:02:05.147017 27034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:05.147086 27034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:02:05.147136 27034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:05.157127 27034 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:37815
I20260929 18:02:05.157228 27162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:37815 every 8 connection(s)
I20260929 18:02:05.158169 27034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 18:02:05.167698 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 27034
I20260929 18:02:05.167824 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance
I20260929 18:02:05.168181 7640 external_mini_cluster.cc:1428] Setting key 5f59d9d313b5d5d79f7c5d51cf2332e0
Sep 29 18:02:05 dist-test-slave-r7ph krb5kdc[26793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704925, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.2@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:02:05.175585 26877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:05.160247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:43511 (local address 127.7.118.62:33971)
0929 18:02:05.160461 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:05.160466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:05.161509 (+ 1043us) server_negotiation.cc:407] Connection header received
0929 18:02:05.162466 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:05.162469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:05.162524 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:05.162622 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:05.164225 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.164782 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.166229 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.166566 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.166669 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:05.170085 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:05.170089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:05.170092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:05.170126 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:05.172342 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:05.173229 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:05.173233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:05.173235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:05.173337 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:05.174032 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:05.174035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:05.174036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:05.174374 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:05.174467 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:05.175305 (+ 838us) server_negotiation.cc:299] Negotiation successful
0929 18:02:05.175452 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260929 18:02:05.177002 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:05.160742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33971 (local address 127.7.118.2:43511)
0929 18:02:05.161324 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:05.161375 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:05.162223 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:05.162866 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:05.162875 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:05.163376 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:05.163969 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:05.163987 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.164934 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.164938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:05.166057 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:05.166068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.166757 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.166761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:05.166937 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:05.167744 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:02:05.167807 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:05.169853 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:05.172501 (+ 2648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:05.172513 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:05.172536 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:05.173040 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:05.173478 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:05.173484 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:05.173489 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:05.173748 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:05.174515 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:05.174526 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:05.174948 (+ 422us) client_negotiation.cc:769] Sending connection context
0929 18:02:05.175210 (+ 262us) client_negotiation.cc:240] Negotiation successful
0929 18:02:05.175678 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":136,"threads_started":1}
I20260929 18:02:05.178661 27163 heartbeater.cc:347] Connected to a master server at 127.7.118.62:33971
I20260929 18:02:05.178952 27163 heartbeater.cc:464] Registering TS with master...
I20260929 18:02:05.179459 27163 heartbeater.cc:511] Master 127.7.118.62:33971 requested a full tablet report, sending...
I20260929 18:02:05.180730 26838 ts_manager.cc:194] Registered new tserver with Master: ac5ae078ef7842cfb1cc7d0ce63c83a3 (127.7.118.2:37815)
WARNING: no policy specified for kudu/127.7.118.3@KRBTEST.COM; defaulting to no policy
I20260929 18:02:05.181433 26838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.2@KRBTEST.COM'} at 127.7.118.2:43511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
Entry for principal kudu/127.7.118.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab.
I20260929 18:02:05.202733 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:0
--local_ip_for_outbound_sockets=127.7.118.3
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:33971
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:05.322788 27168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:05.323141 27168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:05.323222 27168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:05.328603 27168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:05.328760 27168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:05.328881 27168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 18:02:05.335178 27168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.3.keytab
--principal=kudu/127.7.118.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:33971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:05.336440 27168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:05.337602 27168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:05.352725 27173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:05.352721 27176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:05.352872 27174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:05.352865 27168 server_base.cc:1034] running on GCE node
I20260929 18:02:05.353641 27168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:05.354519 27168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:05.355728 27168 hybrid_clock.cc:648] HybridClock initialized: now 1790704925355709 us; error 46 us; skew 500 ppm
Sep 29 18:02:05 dist-test-slave-r7ph krb5kdc[26793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704925, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:05.358923 27168 init.cc:377] Logged in from keytab as kudu/127.7.118.3@KRBTEST.COM (short username kudu)
I20260929 18:02:05.360049 27168 webserver.cc:460] Webserver started at http://127.7.118.3:46233/ using document root <none> and password file <none>
I20260929 18:02:05.360687 27168 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:05.360790 27168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:05.361038 27168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:05.363433 27168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-2/data/instance:
uuid: "92e6372e803445d4be397ce3a3b1ffd8"
format_stamp: "Formatted at 2026-09-29 18:02:05 on dist-test-slave-r7ph"
server_key: "59b5d636ccc6365f799eac7fdc1718d3"
server_key_iv: "58a97594c3e1febb97ffa77d2b65332e"
server_key_version: "encryptionkey@0"
I20260929 18:02:05.364092 27168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance:
uuid: "92e6372e803445d4be397ce3a3b1ffd8"
format_stamp: "Formatted at 2026-09-29 18:02:05 on dist-test-slave-r7ph"
server_key: "59b5d636ccc6365f799eac7fdc1718d3"
server_key_iv: "58a97594c3e1febb97ffa77d2b65332e"
server_key_version: "encryptionkey@0"
I20260929 18:02:05.368479 27168 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 18:02:05.371587 27183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:05.373199 27168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:02:05.373428 27168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "92e6372e803445d4be397ce3a3b1ffd8"
format_stamp: "Formatted at 2026-09-29 18:02:05 on dist-test-slave-r7ph"
server_key: "59b5d636ccc6365f799eac7fdc1718d3"
server_key_iv: "58a97594c3e1febb97ffa77d2b65332e"
server_key_version: "encryptionkey@0"
I20260929 18:02:05.373581 27168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:05.396313 27168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:05.400612 27168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:05.400875 27168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:05.401556 27168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:02:05.402541 27168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:02:05.402616 27168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:05.402690 27168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:02:05.402738 27168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:05.412220 27168 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:43025
I20260929 18:02:05.412287 27296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:43025 every 8 connection(s)
I20260929 18:02:05.413236 27168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 18:02:05.420568 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 27168
I20260929 18:02:05.420756 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance
I20260929 18:02:05.421132 7640 external_mini_cluster.cc:1428] Setting key 739ffc1ce6ec1c7553b48655f63d32f9
Sep 29 18:02:05 dist-test-slave-r7ph krb5kdc[26793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704925, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.3@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:02:05.428112 26877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:05.415526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:55621 (local address 127.7.118.62:33971)
0929 18:02:05.415735 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:05.415740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:05.416695 (+ 955us) server_negotiation.cc:407] Connection header received
0929 18:02:05.417656 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:05.417659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:05.417710 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:05.417789 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:05.419109 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.419666 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.421028 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.421388 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.421511 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:05.424197 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:05.424200 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:05.424202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:05.424229 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:05.425833 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:05.426563 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:05.426567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:05.426569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:05.426662 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:05.427040 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:05.427043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:05.427044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:05.427265 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:05.427343 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:05.427870 (+ 527us) server_negotiation.cc:299] Negotiation successful
0929 18:02:05.427987 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260929 18:02:05.429057 27299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:05.415991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:33971 (local address 127.7.118.3:55621)
0929 18:02:05.416521 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:05.416569 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:05.417430 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:05.417979 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:05.417987 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:05.418415 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:05.418956 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:05.418971 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.419806 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.419809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:05.420901 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:05.420911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.421592 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.421598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:05.421829 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:05.422524 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:02:05.422541 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:05.424032 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:05.425983 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:05.425989 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:05.426005 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:05.426465 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:05.426775 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:05.426778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:05.426780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:05.426939 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:05.427360 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:05.427365 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:05.427630 (+ 265us) client_negotiation.cc:769] Sending connection context
0929 18:02:05.427792 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 18:02:05.428081 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":131,"threads_started":1}
I20260929 18:02:05.430500 27297 heartbeater.cc:347] Connected to a master server at 127.7.118.62:33971
I20260929 18:02:05.430815 27297 heartbeater.cc:464] Registering TS with master...
I20260929 18:02:05.431474 27297 heartbeater.cc:511] Master 127.7.118.62:33971 requested a full tablet report, sending...
I20260929 18:02:05.432687 26838 ts_manager.cc:194] Registered new tserver with Master: 92e6372e803445d4be397ce3a3b1ffd8 (127.7.118.3:43025)
I20260929 18:02:05.433336 26838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.118.3@KRBTEST.COM'} at 127.7.118.3:55621
I20260929 18:02:05.435385 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:02:05.447235 26877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:05.437861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42552 (local address 127.7.118.62:33971)
0929 18:02:05.438053 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:05.438058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:05.438204 (+ 146us) server_negotiation.cc:407] Connection header received
0929 18:02:05.438394 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:05.438397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:05.438449 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:05.438539 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:05.439292 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.439870 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.440802 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.441039 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.441159 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:05.443054 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:05.443058 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:05.443060 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:05.443094 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:05.444860 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:05.445486 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:05.445490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:05.445492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:05.445618 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:05.446128 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:05.446132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:05.446133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:05.446454 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:05.446609 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:05.446910 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 18:02:05.447039 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 18:02:05.450744 26838 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:02:05.453920 26838 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:02:05.470084 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:05.464789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44470 (local address 127.7.118.3:43025)
0929 18:02:05.465114 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:05.465124 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:05.465155 (+ 31us) server_negotiation.cc:407] Connection header received
0929 18:02:05.465226 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:05.465232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:05.465399 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:05.465879 (+ 480us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:02:05.466021 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.467307 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.468414 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.469130 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.469265 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:05.469435 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:05.469759 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 18:02:05.469878 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":114,"threads_started":1}
I20260929 18:02:05.470083 27311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:05.463593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:39939 (local address 127.0.0.1:59000)
0929 18:02:05.464520 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:05.464581 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:05.465095 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:05.465564 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:05.465566 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:05.465585 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:02:05.465816 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:05.465822 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.467041 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.467044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:05.468200 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:05.468211 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.469412 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.469414 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:05.469521 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:05.469573 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 18:02:05.469697 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 18:02:05.469888 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":640,"spinlock_wait_cycles":7296,"thread_start_us":91,"threads_started":1}
I20260929 18:02:05.470451 27312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:05.464466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:43025 (local address 127.0.0.1:44470)
0929 18:02:05.464764 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:05.464776 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:05.464863 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:05.465555 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:05.465558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:05.465587 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:02:05.465728 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:05.465735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.467417 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.467423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:05.468253 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:05.468289 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.469270 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.469273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:05.469596 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:05.469622 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:02:05.469717 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 18:02:05.469893 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":241,"spinlock_wait_cycles":10112,"thread_start_us":81,"threads_started":1}
I20260929 18:02:05.470925 27310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:05.463851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59000 (local address 127.7.118.1:39939)
0929 18:02:05.465109 (+ 1258us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:05.465118 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:05.465155 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:02:05.465235 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:05.465239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:05.465443 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:05.465586 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:02:05.465943 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.466914 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.468346 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.469229 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.469334 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:05.469484 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:05.470633 (+ 1149us) server_negotiation.cc:299] Negotiation successful
0929 18:02:05.470764 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1154,"thread_start_us":77,"threads_started":1}
I20260929 18:02:05.471992 27308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:05.463593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:37815 (local address 127.0.0.1:40764)
0929 18:02:05.464521 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:05.464582 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:05.464792 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:05.465378 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:05.465384 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:05.465405 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:02:05.465572 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:05.465579 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.467137 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.467140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:05.469444 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:05.469454 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.470917 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.470921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:05.471630 (+ 709us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:05.471662 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 18:02:05.471774 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 18:02:05.471860 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"mutex_wait_us":1057,"spinlock_wait_cycles":768,"thread_start_us":109,"threads_started":1}
I20260929 18:02:05.472452 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:05.463768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40764 (local address 127.7.118.2:37815)
0929 18:02:05.464246 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:05.464258 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:05.464673 (+ 415us) server_negotiation.cc:407] Connection header received
0929 18:02:05.464967 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:05.464973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:05.465213 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:05.465375 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:02:05.465711 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.466971 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.469560 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.470524 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.470640 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:05.470847 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:05.472188 (+ 1341us) server_negotiation.cc:299] Negotiation successful
0929 18:02:05.472301 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":111,"threads_started":1}
I20260929 18:02:05.473770 26963 tablet_service.cc:1467] Processing CreateTablet for tablet 1b805017165a4c9b86e21cf25bc1761e (DEFAULT_TABLE table=test-table [id=ac5b38b8aa53488e8147eb1131fce04d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:02:05.473793 27231 tablet_service.cc:1467] Processing CreateTablet for tablet 1b805017165a4c9b86e21cf25bc1761e (DEFAULT_TABLE table=test-table [id=ac5b38b8aa53488e8147eb1131fce04d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:02:05.474975 26963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b805017165a4c9b86e21cf25bc1761e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:05.474978 27231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b805017165a4c9b86e21cf25bc1761e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:05.477666 27097 tablet_service.cc:1467] Processing CreateTablet for tablet 1b805017165a4c9b86e21cf25bc1761e (DEFAULT_TABLE table=test-table [id=ac5b38b8aa53488e8147eb1131fce04d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:02:05.479027 27097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b805017165a4c9b86e21cf25bc1761e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:05.482348 27314 tablet_bootstrap.cc:492] T 1b805017165a4c9b86e21cf25bc1761e P 92e6372e803445d4be397ce3a3b1ffd8: Bootstrap starting.
I20260929 18:02:05.483742 27315 tablet_bootstrap.cc:492] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c: Bootstrap starting.
I20260929 18:02:05.485177 27314 tablet_bootstrap.cc:654] T 1b805017165a4c9b86e21cf25bc1761e P 92e6372e803445d4be397ce3a3b1ffd8: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:05.486063 27315 tablet_bootstrap.cc:654] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:05.487015 27315 log.cc:824] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:05.487015 27314 log.cc:824] T 1b805017165a4c9b86e21cf25bc1761e P 92e6372e803445d4be397ce3a3b1ffd8: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:05.487514 27316 tablet_bootstrap.cc:492] T 1b805017165a4c9b86e21cf25bc1761e P ac5ae078ef7842cfb1cc7d0ce63c83a3: Bootstrap starting.
I20260929 18:02:05.489364 27315 tablet_bootstrap.cc:492] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c: No bootstrap required, opened a new log
I20260929 18:02:05.489364 27314 tablet_bootstrap.cc:492] T 1b805017165a4c9b86e21cf25bc1761e P 92e6372e803445d4be397ce3a3b1ffd8: No bootstrap required, opened a new log
I20260929 18:02:05.489593 27315 ts_tablet_manager.cc:1397] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:02:05.489593 27314 ts_tablet_manager.cc:1397] T 1b805017165a4c9b86e21cf25bc1761e P 92e6372e803445d4be397ce3a3b1ffd8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 18:02:05.489939 27316 tablet_bootstrap.cc:654] T 1b805017165a4c9b86e21cf25bc1761e P ac5ae078ef7842cfb1cc7d0ce63c83a3: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:05.490787 27316 log.cc:824] T 1b805017165a4c9b86e21cf25bc1761e P ac5ae078ef7842cfb1cc7d0ce63c83a3: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:05.492745 27316 tablet_bootstrap.cc:492] T 1b805017165a4c9b86e21cf25bc1761e P ac5ae078ef7842cfb1cc7d0ce63c83a3: No bootstrap required, opened a new log
I20260929 18:02:05.492977 27316 ts_tablet_manager.cc:1397] T 1b805017165a4c9b86e21cf25bc1761e P ac5ae078ef7842cfb1cc7d0ce63c83a3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:02:05.494525 27315 raft_consensus.cc:348] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac5ae078ef7842cfb1cc7d0ce63c83a3" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 37815 } } peers { permanent_uuid: "c2c613a0dedb4402b485e679fe5f6a0c" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 39939 } } peers { permanent_uuid: "92e6372e803445d4be397ce3a3b1ffd8" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 43025 } }
I20260929 18:02:05.494877 27315 raft_consensus.cc:374] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:05.494953 27315 raft_consensus.cc:729] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2c613a0dedb4402b485e679fe5f6a0c, State: Initialized, Role: FOLLOWER
I20260929 18:02:05.495512 27315 consensus_queue.cc:260] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac5ae078ef7842cfb1cc7d0ce63c83a3" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 37815 } } peers { permanent_uuid: "c2c613a0dedb4402b485e679fe5f6a0c" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 39939 } } peers { permanent_uuid: "92e6372e803445d4be397ce3a3b1ffd8" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 43025 } }
I20260929 18:02:05.496496 27029 heartbeater.cc:503] Master 127.7.118.62:33971 was elected leader, sending a full tablet report...
I20260929 18:02:05.496827 27315 ts_tablet_manager.cc:1428] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260929 18:02:05.497345 27316 raft_consensus.cc:348] T 1b805017165a4c9b86e21cf25bc1761e P ac5ae078ef7842cfb1cc7d0ce63c83a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac5ae078ef7842cfb1cc7d0ce63c83a3" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 37815 } } peers { permanent_uuid: "c2c613a0dedb4402b485e679fe5f6a0c" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 39939 } } peers { permanent_uuid: "92e6372e803445d4be397ce3a3b1ffd8" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 43025 } }
I20260929 18:02:05.497668 27316 raft_consensus.cc:374] T 1b805017165a4c9b86e21cf25bc1761e P ac5ae078ef7842cfb1cc7d0ce63c83a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:05.497727 27316 raft_consensus.cc:729] T 1b805017165a4c9b86e21cf25bc1761e P ac5ae078ef7842cfb1cc7d0ce63c83a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac5ae078ef7842cfb1cc7d0ce63c83a3, State: Initialized, Role: FOLLOWER
I20260929 18:02:05.497620 27314 raft_consensus.cc:348] T 1b805017165a4c9b86e21cf25bc1761e P 92e6372e803445d4be397ce3a3b1ffd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac5ae078ef7842cfb1cc7d0ce63c83a3" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 37815 } } peers { permanent_uuid: "c2c613a0dedb4402b485e679fe5f6a0c" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 39939 } } peers { permanent_uuid: "92e6372e803445d4be397ce3a3b1ffd8" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 43025 } }
I20260929 18:02:05.497956 27314 raft_consensus.cc:374] T 1b805017165a4c9b86e21cf25bc1761e P 92e6372e803445d4be397ce3a3b1ffd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:05.498019 27314 raft_consensus.cc:729] T 1b805017165a4c9b86e21cf25bc1761e P 92e6372e803445d4be397ce3a3b1ffd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e6372e803445d4be397ce3a3b1ffd8, State: Initialized, Role: FOLLOWER
I20260929 18:02:05.498283 27316 consensus_queue.cc:260] T 1b805017165a4c9b86e21cf25bc1761e P ac5ae078ef7842cfb1cc7d0ce63c83a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac5ae078ef7842cfb1cc7d0ce63c83a3" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 37815 } } peers { permanent_uuid: "c2c613a0dedb4402b485e679fe5f6a0c" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 39939 } } peers { permanent_uuid: "92e6372e803445d4be397ce3a3b1ffd8" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 43025 } }
I20260929 18:02:05.498490 27314 consensus_queue.cc:260] T 1b805017165a4c9b86e21cf25bc1761e P 92e6372e803445d4be397ce3a3b1ffd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac5ae078ef7842cfb1cc7d0ce63c83a3" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 37815 } } peers { permanent_uuid: "c2c613a0dedb4402b485e679fe5f6a0c" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 39939 } } peers { permanent_uuid: "92e6372e803445d4be397ce3a3b1ffd8" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 43025 } }
I20260929 18:02:05.499409 27316 ts_tablet_manager.cc:1428] T 1b805017165a4c9b86e21cf25bc1761e P ac5ae078ef7842cfb1cc7d0ce63c83a3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 18:02:05.499454 27297 heartbeater.cc:503] Master 127.7.118.62:33971 was elected leader, sending a full tablet report...
I20260929 18:02:05.499528 27314 ts_tablet_manager.cc:1428] T 1b805017165a4c9b86e21cf25bc1761e P 92e6372e803445d4be397ce3a3b1ffd8: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260929 18:02:05.499773 27163 heartbeater.cc:503] Master 127.7.118.62:33971 was elected leader, sending a full tablet report...
W20260929 18:02:05.659497 27164 tablet.cc:2369] T 1b805017165a4c9b86e21cf25bc1761e P ac5ae078ef7842cfb1cc7d0ce63c83a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:02:05.664351 27298 tablet.cc:2369] T 1b805017165a4c9b86e21cf25bc1761e P 92e6372e803445d4be397ce3a3b1ffd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:02:05.668880 27030 tablet.cc:2369] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:02:05.759564 27320 raft_consensus.cc:482] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:02:05.759829 27320 raft_consensus.cc:504] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac5ae078ef7842cfb1cc7d0ce63c83a3" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 37815 } } peers { permanent_uuid: "c2c613a0dedb4402b485e679fe5f6a0c" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 39939 } } peers { permanent_uuid: "92e6372e803445d4be397ce3a3b1ffd8" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 43025 } }
I20260929 18:02:05.761318 27320 leader_election.cc:288] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac5ae078ef7842cfb1cc7d0ce63c83a3 (127.7.118.2:37815), 92e6372e803445d4be397ce3a3b1ffd8 (127.7.118.3:43025)
I20260929 18:02:05.765501 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:05.761705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:40589 (local address 127.7.118.2:37815)
0929 18:02:05.761849 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:05.761853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:05.762284 (+ 431us) server_negotiation.cc:407] Connection header received
0929 18:02:05.762380 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:05.762383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:05.762438 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:05.762553 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:02:05.762863 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.763404 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.764522 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.764921 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.765040 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:05.765106 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:05.765286 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 18:02:05.765363 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:02:05.765512 27323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:05.761589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:37815 (local address 127.7.118.1:40589)
0929 18:02:05.761922 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:05.761941 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:05.762091 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:05.762570 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:05.762574 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:05.762594 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:02:05.762742 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:05.762747 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.763522 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.763528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:05.764387 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:05.764397 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.765046 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.765050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:05.765150 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:05.765179 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 18:02:05.765270 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 18:02:05.765349 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":152,"threads_started":1}
I20260929 18:02:05.766219 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:05.762377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:38371 (local address 127.7.118.3:43025)
0929 18:02:05.762539 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:05.762545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:05.762623 (+ 78us) server_negotiation.cc:407] Connection header received
0929 18:02:05.762813 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:05.762818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:05.762869 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:05.762964 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:02:05.763254 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.763797 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.764962 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.765409 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.765543 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:05.765606 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:05.766014 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 18:02:05.766105 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 18:02:05.766230 27324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:05.762079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:43025 (local address 127.7.118.1:38371)
0929 18:02:05.762528 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:05.762553 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:05.762681 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:05.762977 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:05.762980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:05.762998 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:02:05.763122 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:05.763131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.763919 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.763924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:05.764816 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:05.764825 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.765563 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.765567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:05.765680 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:05.765701 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:02:05.765984 (+ 283us) client_negotiation.cc:240] Negotiation successful
0929 18:02:05.766063 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"spinlock_wait_cycles":1920,"thread_start_us":109,"threads_started":1}
I20260929 18:02:05.766438 27117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b805017165a4c9b86e21cf25bc1761e" candidate_uuid: "c2c613a0dedb4402b485e679fe5f6a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac5ae078ef7842cfb1cc7d0ce63c83a3" is_pre_election: true
I20260929 18:02:05.766794 27117 raft_consensus.cc:2454] T 1b805017165a4c9b86e21cf25bc1761e P ac5ae078ef7842cfb1cc7d0ce63c83a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2c613a0dedb4402b485e679fe5f6a0c in term 0.
I20260929 18:02:05.766878 27251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b805017165a4c9b86e21cf25bc1761e" candidate_uuid: "c2c613a0dedb4402b485e679fe5f6a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92e6372e803445d4be397ce3a3b1ffd8" is_pre_election: true
I20260929 18:02:05.767233 27251 raft_consensus.cc:2454] T 1b805017165a4c9b86e21cf25bc1761e P 92e6372e803445d4be397ce3a3b1ffd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2c613a0dedb4402b485e679fe5f6a0c in term 0.
I20260929 18:02:05.767343 26919 leader_election.cc:302] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac5ae078ef7842cfb1cc7d0ce63c83a3, c2c613a0dedb4402b485e679fe5f6a0c; no voters:
I20260929 18:02:05.767699 27320 raft_consensus.cc:2781] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:02:05.767803 27320 raft_consensus.cc:482] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:02:05.767863 27320 raft_consensus.cc:3037] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:05.768916 27320 raft_consensus.cc:504] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac5ae078ef7842cfb1cc7d0ce63c83a3" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 37815 } } peers { permanent_uuid: "c2c613a0dedb4402b485e679fe5f6a0c" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 39939 } } peers { permanent_uuid: "92e6372e803445d4be397ce3a3b1ffd8" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 43025 } }
I20260929 18:02:05.769284 27320 leader_election.cc:288] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c [CANDIDATE]: Term 1 election: Requested vote from peers ac5ae078ef7842cfb1cc7d0ce63c83a3 (127.7.118.2:37815), 92e6372e803445d4be397ce3a3b1ffd8 (127.7.118.3:43025)
I20260929 18:02:05.769866 27117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b805017165a4c9b86e21cf25bc1761e" candidate_uuid: "c2c613a0dedb4402b485e679fe5f6a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac5ae078ef7842cfb1cc7d0ce63c83a3"
I20260929 18:02:05.769877 27251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b805017165a4c9b86e21cf25bc1761e" candidate_uuid: "c2c613a0dedb4402b485e679fe5f6a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92e6372e803445d4be397ce3a3b1ffd8"
I20260929 18:02:05.770032 27117 raft_consensus.cc:3037] T 1b805017165a4c9b86e21cf25bc1761e P ac5ae078ef7842cfb1cc7d0ce63c83a3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:05.770051 27251 raft_consensus.cc:3037] T 1b805017165a4c9b86e21cf25bc1761e P 92e6372e803445d4be397ce3a3b1ffd8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:05.771225 27251 raft_consensus.cc:2454] T 1b805017165a4c9b86e21cf25bc1761e P 92e6372e803445d4be397ce3a3b1ffd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2c613a0dedb4402b485e679fe5f6a0c in term 1.
I20260929 18:02:05.771227 27117 raft_consensus.cc:2454] T 1b805017165a4c9b86e21cf25bc1761e P ac5ae078ef7842cfb1cc7d0ce63c83a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2c613a0dedb4402b485e679fe5f6a0c in term 1.
I20260929 18:02:05.771593 26919 leader_election.cc:302] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac5ae078ef7842cfb1cc7d0ce63c83a3, c2c613a0dedb4402b485e679fe5f6a0c; no voters:
I20260929 18:02:05.771813 27320 raft_consensus.cc:2781] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:05.772049 27320 raft_consensus.cc:686] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c [term 1 LEADER]: Becoming Leader. State: Replica: c2c613a0dedb4402b485e679fe5f6a0c, State: Running, Role: LEADER
I20260929 18:02:05.772437 27320 consensus_queue.cc:237] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac5ae078ef7842cfb1cc7d0ce63c83a3" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 37815 } } peers { permanent_uuid: "c2c613a0dedb4402b485e679fe5f6a0c" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 39939 } } peers { permanent_uuid: "92e6372e803445d4be397ce3a3b1ffd8" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 43025 } }
I20260929 18:02:05.776383 26837 catalog_manager.cc:5495] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c reported cstate change: term changed from 0 to 1, leader changed from <none> to c2c613a0dedb4402b485e679fe5f6a0c (127.7.118.1). New cstate: current_term: 1 leader_uuid: "c2c613a0dedb4402b485e679fe5f6a0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac5ae078ef7842cfb1cc7d0ce63c83a3" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2c613a0dedb4402b485e679fe5f6a0c" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 39939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92e6372e803445d4be397ce3a3b1ffd8" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 43025 } health_report { overall_health: UNKNOWN } } }
I20260929 18:02:05.837718 27310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:05.833864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59004 (local address 127.7.118.1:39939)
0929 18:02:05.834008 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:05.834014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:05.834043 (+ 29us) server_negotiation.cc:407] Connection header received
0929 18:02:05.834185 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:05.834188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:05.834243 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:05.834388 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:02:05.834674 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.835201 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.836141 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:05.836363 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:05.836475 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:05.836807 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:02:05.837319 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:02:05.837459 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:05.837502 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 18:02:05.837565 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 18:02:05.844441 27117 raft_consensus.cc:1261] T 1b805017165a4c9b86e21cf25bc1761e P ac5ae078ef7842cfb1cc7d0ce63c83a3 [term 1 FOLLOWER]: Refusing update from remote peer c2c613a0dedb4402b485e679fe5f6a0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:02:05.844441 27251 raft_consensus.cc:1261] T 1b805017165a4c9b86e21cf25bc1761e P 92e6372e803445d4be397ce3a3b1ffd8 [term 1 FOLLOWER]: Refusing update from remote peer c2c613a0dedb4402b485e679fe5f6a0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:02:05.845371 27325 consensus_queue.cc:1035] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac5ae078ef7842cfb1cc7d0ce63c83a3" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:02:05.845644 27320 consensus_queue.cc:1035] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "92e6372e803445d4be397ce3a3b1ffd8" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 43025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:02:05.854349 27330 mvcc.cc:205] Tried to move back new op lower bound from 7334727376250073088 to 7334727375966994432. Current Snapshot: MvccSnapshot[applied={T|T < 7334727376250073088}]
I20260929 18:02:05.855093 27329 mvcc.cc:205] Tried to move back new op lower bound from 7334727376250073088 to 7334727375966994432. Current Snapshot: MvccSnapshot[applied={T|T < 7334727376250073088}]
I20260929 18:02:05.863878 26836 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:02:05.864049 26836 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:02:05.866844 26836 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b6b023f9c1a42b29a3a5d75e18c3848: Sending DeleteTablet for 3 replicas of tablet 1b805017165a4c9b86e21cf25bc1761e
I20260929 18:02:05.868026 27097 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b805017165a4c9b86e21cf25bc1761e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:02:05 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:40764
I20260929 18:02:05.868036 26963 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b805017165a4c9b86e21cf25bc1761e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:02:05 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:59000
I20260929 18:02:05.868149 27231 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b805017165a4c9b86e21cf25bc1761e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:02:05 UTC) from {username='kudu', principal='kudu/127.7.118.62@KRBTEST.COM'} at 127.0.0.1:44470
I20260929 18:02:05.868592 27338 tablet_replica.cc:330] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c: stopping tablet replica
I20260929 18:02:05.868664 27340 tablet_replica.cc:330] T 1b805017165a4c9b86e21cf25bc1761e P 92e6372e803445d4be397ce3a3b1ffd8: stopping tablet replica
I20260929 18:02:05.868963 27339 tablet_replica.cc:330] T 1b805017165a4c9b86e21cf25bc1761e P ac5ae078ef7842cfb1cc7d0ce63c83a3: stopping tablet replica
I20260929 18:02:05.868969 27338 raft_consensus.cc:2229] T 1b805017165a4c9b86e21cf25bc1761e P c2c613a0dedb4402b485e679fe5f6a0c [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:02:05.869010 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 26900
I20260929 18:02:05.869354 27339 raft_consensus.cc:2229] T 1b805017165a4c9b86e21cf25bc1761e P ac5ae078ef7842cfb1cc7d0ce63c83a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:02:05.869758 27339 raft_consensus.cc:2258] T 1b805017165a4c9b86e21cf25bc1761e P ac5ae078ef7842cfb1cc7d0ce63c83a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:02:05.871112 27340 raft_consensus.cc:2229] T 1b805017165a4c9b86e21cf25bc1761e P 92e6372e803445d4be397ce3a3b1ffd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:02:05.871471 27340 raft_consensus.cc:2258] T 1b805017165a4c9b86e21cf25bc1761e P 92e6372e803445d4be397ce3a3b1ffd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:02:05.872627 27339 ts_tablet_manager.cc:1905] T 1b805017165a4c9b86e21cf25bc1761e P ac5ae078ef7842cfb1cc7d0ce63c83a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:02:05.875214 27340 ts_tablet_manager.cc:1905] T 1b805017165a4c9b86e21cf25bc1761e P 92e6372e803445d4be397ce3a3b1ffd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:02:05.875492 27339 ts_tablet_manager.cc:1918] T 1b805017165a4c9b86e21cf25bc1761e P ac5ae078ef7842cfb1cc7d0ce63c83a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:02:05.875613 27339 log.cc:1196] T 1b805017165a4c9b86e21cf25bc1761e P ac5ae078ef7842cfb1cc7d0ce63c83a3: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-1/wal/wals/1b805017165a4c9b86e21cf25bc1761e
I20260929 18:02:05.875980 27339 ts_tablet_manager.cc:1939] T 1b805017165a4c9b86e21cf25bc1761e P ac5ae078ef7842cfb1cc7d0ce63c83a3: Deleting consensus metadata
W20260929 18:02:05.876585 26824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.118.1:39939 (error 108)
I20260929 18:02:05.876809 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 27034
I20260929 18:02:05.879333 27340 ts_tablet_manager.cc:1918] T 1b805017165a4c9b86e21cf25bc1761e P 92e6372e803445d4be397ce3a3b1ffd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:02:05.879472 27340 log.cc:1196] T 1b805017165a4c9b86e21cf25bc1761e P 92e6372e803445d4be397ce3a3b1ffd8: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704657566865-7640-0/minicluster-data/ts-2/wal/wals/1b805017165a4c9b86e21cf25bc1761e
I20260929 18:02:05.879509 27308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:05.879094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:39939 (local address 127.0.0.1:59018)
0929 18:02:05.879316 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:05.879398 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.1:39939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260929 18:02:05.879721 26824 catalog_manager.cc:4584] TS c2c613a0dedb4402b485e679fe5f6a0c (127.7.118.1:39939): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1b805017165a4c9b86e21cf25bc1761e: Network error: Client connection negotiation failed: client connection to 127.7.118.1:39939: connect: Connection refused (error 111)
I20260929 18:02:05.879819 27340 ts_tablet_manager.cc:1939] T 1b805017165a4c9b86e21cf25bc1761e P 92e6372e803445d4be397ce3a3b1ffd8: Deleting consensus metadata
I20260929 18:02:05.880998 26825 catalog_manager.cc:4841] TS 92e6372e803445d4be397ce3a3b1ffd8 (127.7.118.3:43025): tablet 1b805017165a4c9b86e21cf25bc1761e (table test-table [id=ac5b38b8aa53488e8147eb1131fce04d]) successfully deleted
I20260929 18:02:05.883409 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 27168
I20260929 18:02:05.884757 27308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:05.884396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:37815 (local address 127.0.0.1:40778)
0929 18:02:05.884540 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:05.884645 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.2:37815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 18:02:05.889488 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 26806
2026-09-29T18:02:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2695 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T18:02:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:02:05Z Disabled control of system clock
I20260929 18:02:05.921669 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:37631
--webserver_interface=127.7.118.62
--webserver_port=0
--builtin_ntp_servers=127.7.118.20:34661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:37631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:02:06.044531 27345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:06.044862 27345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:06.044912 27345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:06.050401 27345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:06.050495 27345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:06.050511 27345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:06.050529 27345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:06.050545 27345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:06.055909 27345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:34661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:37631
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:37631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:06.057147 27345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:06.058722 27345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:06.068591 27350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:06.068681 27351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:06.068681 27353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:06.068821 27345 server_base.cc:1034] running on GCE node
I20260929 18:02:06.069566 27345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:06.070974 27345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:06.072223 27345 hybrid_clock.cc:648] HybridClock initialized: now 1790704926072192 us; error 52 us; skew 500 ppm
I20260929 18:02:06.082916 27345 webserver.cc:460] Webserver started at http://127.7.118.62:35371/ using document root <none> and password file <none>
I20260929 18:02:06.083607 27345 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:06.083686 27345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:06.083875 27345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:06.086221 27345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "e90f41d100234b8e886eba7281c4d094"
format_stamp: "Formatted at 2026-09-29 18:02:06 on dist-test-slave-r7ph"
server_key: "336a80c2d95cb5e7806711ecc750c902"
server_key_iv: "87c60a2e3454dfb06fefacdb7d0d01e8"
server_key_version: "encryptionkey@0"
I20260929 18:02:06.086920 27345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "e90f41d100234b8e886eba7281c4d094"
format_stamp: "Formatted at 2026-09-29 18:02:06 on dist-test-slave-r7ph"
server_key: "336a80c2d95cb5e7806711ecc750c902"
server_key_iv: "87c60a2e3454dfb06fefacdb7d0d01e8"
server_key_version: "encryptionkey@0"
I20260929 18:02:06.091339 27345 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:02:06.094431 27359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:06.095992 27345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:02:06.096132 27345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "e90f41d100234b8e886eba7281c4d094"
format_stamp: "Formatted at 2026-09-29 18:02:06 on dist-test-slave-r7ph"
server_key: "336a80c2d95cb5e7806711ecc750c902"
server_key_iv: "87c60a2e3454dfb06fefacdb7d0d01e8"
server_key_version: "encryptionkey@0"
I20260929 18:02:06.096251 27345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:06.120100 27345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:06.120872 27345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:06.121040 27345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:06.129057 27345 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:37631
I20260929 18:02:06.129182 27411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:37631 every 8 connection(s)
I20260929 18:02:06.130232 27345 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:02:06.134050 27412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:06.139587 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 27345
I20260929 18:02:06.139780 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:02:06.140079 7640 external_mini_cluster.cc:1428] Setting key 1940aae8f3769fcdaa4d3bc6ed7ae328
I20260929 18:02:06.141016 27412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90f41d100234b8e886eba7281c4d094: Bootstrap starting.
I20260929 18:02:06.143715 27412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e90f41d100234b8e886eba7281c4d094: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:06.144798 27412 log.cc:824] T 00000000000000000000000000000000 P e90f41d100234b8e886eba7281c4d094: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:06.147431 27412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90f41d100234b8e886eba7281c4d094: No bootstrap required, opened a new log
I20260929 18:02:06.151618 27415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:06.141767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33378 (local address 127.7.118.62:37631)
0929 18:02:06.142554 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:06.142570 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:06.142611 (+ 41us) server_negotiation.cc:407] Connection header received
0929 18:02:06.143338 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:06.143358 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:06.143791 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:06.144179 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:06.144623 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.145963 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.147284 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.148086 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.148221 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:06.149305 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:06.149311 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:06.149327 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:06.149850 (+ 523us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:06.149917 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:06.149942 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:06.150085 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:06.150235 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 18:02:06.150691 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":156,"threads_started":1}
I20260929 18:02:06.151957 27412 raft_consensus.cc:348] T 00000000000000000000000000000000 P e90f41d100234b8e886eba7281c4d094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90f41d100234b8e886eba7281c4d094" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 37631 } }
I20260929 18:02:06.152292 27412 raft_consensus.cc:374] T 00000000000000000000000000000000 P e90f41d100234b8e886eba7281c4d094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:06.152348 27412 raft_consensus.cc:729] T 00000000000000000000000000000000 P e90f41d100234b8e886eba7281c4d094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e90f41d100234b8e886eba7281c4d094, State: Initialized, Role: FOLLOWER
I20260929 18:02:06.152848 27412 consensus_queue.cc:260] T 00000000000000000000000000000000 P e90f41d100234b8e886eba7281c4d094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90f41d100234b8e886eba7281c4d094" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 37631 } }
I20260929 18:02:06.153019 27412 raft_consensus.cc:388] T 00000000000000000000000000000000 P e90f41d100234b8e886eba7281c4d094 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:02:06.153108 27412 raft_consensus.cc:482] T 00000000000000000000000000000000 P e90f41d100234b8e886eba7281c4d094 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:02:06.153246 27412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e90f41d100234b8e886eba7281c4d094 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:06.154393 27412 raft_consensus.cc:504] T 00000000000000000000000000000000 P e90f41d100234b8e886eba7281c4d094 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90f41d100234b8e886eba7281c4d094" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 37631 } }
I20260929 18:02:06.154776 27412 leader_election.cc:302] T 00000000000000000000000000000000 P e90f41d100234b8e886eba7281c4d094 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e90f41d100234b8e886eba7281c4d094; no voters:
I20260929 18:02:06.155135 27412 leader_election.cc:288] T 00000000000000000000000000000000 P e90f41d100234b8e886eba7281c4d094 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:02:06.155301 27417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e90f41d100234b8e886eba7281c4d094 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:06.155608 27417 raft_consensus.cc:686] T 00000000000000000000000000000000 P e90f41d100234b8e886eba7281c4d094 [term 1 LEADER]: Becoming Leader. State: Replica: e90f41d100234b8e886eba7281c4d094, State: Running, Role: LEADER
I20260929 18:02:06.156003 27417 consensus_queue.cc:237] T 00000000000000000000000000000000 P e90f41d100234b8e886eba7281c4d094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90f41d100234b8e886eba7281c4d094" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 37631 } }
I20260929 18:02:06.156596 27412 sys_catalog.cc:564] T 00000000000000000000000000000000 P e90f41d100234b8e886eba7281c4d094 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:02:06.157963 27419 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90f41d100234b8e886eba7281c4d094 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e90f41d100234b8e886eba7281c4d094. Latest consensus state: current_term: 1 leader_uuid: "e90f41d100234b8e886eba7281c4d094" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90f41d100234b8e886eba7281c4d094" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 37631 } } }
I20260929 18:02:06.158004 27418 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90f41d100234b8e886eba7281c4d094 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e90f41d100234b8e886eba7281c4d094" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90f41d100234b8e886eba7281c4d094" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 37631 } } }
I20260929 18:02:06.158130 27419 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90f41d100234b8e886eba7281c4d094 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:06.158129 27418 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90f41d100234b8e886eba7281c4d094 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:06.158676 27428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:02:06.162638 27428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:02:06.169579 27428 catalog_manager.cc:1347] Generated new cluster ID: e29e695adea04f599d3dd24be8717086
I20260929 18:02:06.169679 27428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:02:06.199522 27428 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:02:06.200744 27428 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:02:06.211344 27428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e90f41d100234b8e886eba7281c4d094: Generated new TSK 0
I20260929 18:02:06.212312 27428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:02:06.221076 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.1:0
--local_ip_for_outbound_sockets=127.7.118.1
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:37631
--builtin_ntp_servers=127.7.118.20:34661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:02:06.352066 27436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:06.352389 27436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:06.352432 27436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:06.356957 27436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:06.357043 27436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:06.357132 27436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.1
I20260929 18:02:06.362701 27436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:34661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.118.1
--webserver_port=0
--tserver_master_addrs=127.7.118.62:37631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.1
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:06.363919 27436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:06.365433 27436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:06.376236 27444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:06.376389 27436 server_base.cc:1034] running on GCE node
W20260929 18:02:06.376237 27441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:06.376715 27442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:06.377342 27436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:06.378108 27436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:06.379333 27436 hybrid_clock.cc:648] HybridClock initialized: now 1790704926379313 us; error 38 us; skew 500 ppm
I20260929 18:02:06.381690 27436 webserver.cc:460] Webserver started at http://127.7.118.1:39521/ using document root <none> and password file <none>
I20260929 18:02:06.382475 27436 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:06.382560 27436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:06.382793 27436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:06.385119 27436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-0/data/instance:
uuid: "49af9d4ea55e4d0686ef0f831d19e699"
format_stamp: "Formatted at 2026-09-29 18:02:06 on dist-test-slave-r7ph"
server_key: "b508526a39e7141a5914c6cc864dd9b1"
server_key_iv: "f8c5d78ca520375041496f2b64e868aa"
server_key_version: "encryptionkey@0"
I20260929 18:02:06.385775 27436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance:
uuid: "49af9d4ea55e4d0686ef0f831d19e699"
format_stamp: "Formatted at 2026-09-29 18:02:06 on dist-test-slave-r7ph"
server_key: "b508526a39e7141a5914c6cc864dd9b1"
server_key_iv: "f8c5d78ca520375041496f2b64e868aa"
server_key_version: "encryptionkey@0"
I20260929 18:02:06.390151 27436 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 18:02:06.393286 27450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:06.394930 27436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:02:06.395107 27436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-0/wal
uuid: "49af9d4ea55e4d0686ef0f831d19e699"
format_stamp: "Formatted at 2026-09-29 18:02:06 on dist-test-slave-r7ph"
server_key: "b508526a39e7141a5914c6cc864dd9b1"
server_key_iv: "f8c5d78ca520375041496f2b64e868aa"
server_key_version: "encryptionkey@0"
I20260929 18:02:06.395267 27436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:06.417658 27436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:06.418556 27436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:06.418795 27436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:06.419658 27436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:02:06.420935 27436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:02:06.421013 27436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:06.421084 27436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:02:06.421116 27436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:06.430540 27436 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.1:40203
I20260929 18:02:06.430564 27563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.1:40203 every 8 connection(s)
I20260929 18:02:06.431533 27436 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-0/data/info.pb
I20260929 18:02:06.439371 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 27436
I20260929 18:02:06.439661 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-0/wal/instance
I20260929 18:02:06.440050 7640 external_mini_cluster.cc:1428] Setting key 9f22784013cd3e30733eece6ac67f39b
I20260929 18:02:06.442677 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.2:0
--local_ip_for_outbound_sockets=127.7.118.2
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:37631
--builtin_ntp_servers=127.7.118.20:34661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:02:06.442824 27415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:06.433902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.1:33941 (local address 127.7.118.62:37631)
0929 18:02:06.434106 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:06.434109 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:06.435134 (+ 1025us) server_negotiation.cc:407] Connection header received
0929 18:02:06.436736 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:06.436739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:06.436827 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:06.436943 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:06.437962 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.438602 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.439950 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.440231 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.440348 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:06.442207 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:06.442211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:06.442213 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:06.442226 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:06.442298 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:06.442313 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:06.442407 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:06.442602 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 18:02:06.442676 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:02:06.443863 27566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:06.434104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:37631 (local address 127.7.118.1:33941)
0929 18:02:06.434946 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:06.435004 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:06.436430 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:06.437161 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:06.437170 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:06.437276 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:06.437804 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:06.437818 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.438753 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.438756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:06.439818 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:06.439829 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.440417 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.440423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:06.440682 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:06.441773 (+ 1091us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:02:06.441775 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:06.441851 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:02:06.441958 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:06.442425 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:06.442436 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:06.442442 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:02:06.442592 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 18:02:06.442965 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":176,"threads_started":1}
I20260929 18:02:06.445438 27564 heartbeater.cc:347] Connected to a master server at 127.7.118.62:37631
I20260929 18:02:06.445775 27564 heartbeater.cc:464] Registering TS with master...
I20260929 18:02:06.446622 27564 heartbeater.cc:511] Master 127.7.118.62:37631 requested a full tablet report, sending...
I20260929 18:02:06.448752 27376 ts_manager.cc:194] Registered new tserver with Master: 49af9d4ea55e4d0686ef0f831d19e699 (127.7.118.1:40203)
I20260929 18:02:06.450115 27376 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.118.1:33941
W20260929 18:02:06.570986 27567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:06.571307 27567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:06.571352 27567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:06.576038 27567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:06.576122 27567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:06.576184 27567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.2
I20260929 18:02:06.581507 27567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:34661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.118.2
--webserver_port=0
--tserver_master_addrs=127.7.118.62:37631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.2
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:06.582798 27567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:06.584272 27567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:06.595287 27572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:06.595249 27573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:06.595249 27575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:06.595474 27567 server_base.cc:1034] running on GCE node
I20260929 18:02:06.596076 27567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:06.596834 27567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:06.598055 27567 hybrid_clock.cc:648] HybridClock initialized: now 1790704926598022 us; error 48 us; skew 500 ppm
I20260929 18:02:06.600868 27567 webserver.cc:460] Webserver started at http://127.7.118.2:41979/ using document root <none> and password file <none>
I20260929 18:02:06.601583 27567 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:06.601662 27567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:06.601893 27567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:06.604246 27567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-1/data/instance:
uuid: "703db8732705428f9db94f2458144489"
format_stamp: "Formatted at 2026-09-29 18:02:06 on dist-test-slave-r7ph"
server_key: "a38b755e773565c4b04c5d15c42ad301"
server_key_iv: "1c9317cc210854cc5316faeac652e824"
server_key_version: "encryptionkey@0"
I20260929 18:02:06.604904 27567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance:
uuid: "703db8732705428f9db94f2458144489"
format_stamp: "Formatted at 2026-09-29 18:02:06 on dist-test-slave-r7ph"
server_key: "a38b755e773565c4b04c5d15c42ad301"
server_key_iv: "1c9317cc210854cc5316faeac652e824"
server_key_version: "encryptionkey@0"
I20260929 18:02:06.609254 27567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:02:06.612447 27581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:06.613963 27567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:02:06.614095 27567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-1/wal
uuid: "703db8732705428f9db94f2458144489"
format_stamp: "Formatted at 2026-09-29 18:02:06 on dist-test-slave-r7ph"
server_key: "a38b755e773565c4b04c5d15c42ad301"
server_key_iv: "1c9317cc210854cc5316faeac652e824"
server_key_version: "encryptionkey@0"
I20260929 18:02:06.614322 27567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:06.636981 27567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:06.637912 27567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:06.638191 27567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:06.639160 27567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:02:06.640511 27567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:02:06.640578 27567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:06.640636 27567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:02:06.640699 27567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:06.652061 27567 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.2:34143
I20260929 18:02:06.652151 27694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.2:34143 every 8 connection(s)
I20260929 18:02:06.653407 27567 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-1/data/info.pb
I20260929 18:02:06.661761 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 27567
I20260929 18:02:06.661886 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-1/wal/instance
I20260929 18:02:06.662171 7640 external_mini_cluster.cc:1428] Setting key 89a15f745d1f4fee9a66773fee00f92b
I20260929 18:02:06.664474 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.118.3:0
--local_ip_for_outbound_sockets=127.7.118.3
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:37631
--builtin_ntp_servers=127.7.118.20:34661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:02:06.665971 27415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:06.655726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:37839 (local address 127.7.118.62:37631)
0929 18:02:06.655880 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:06.655884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:06.657442 (+ 1558us) server_negotiation.cc:407] Connection header received
0929 18:02:06.659475 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:06.659479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:06.659551 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:06.659675 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:06.660915 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.661493 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.663193 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.663422 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.663731 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:06.665236 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:06.665243 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:06.665249 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:06.665268 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:06.665298 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:06.665312 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:06.665405 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:06.665642 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 18:02:06.665746 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:02:06.667639 27697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:06.656112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:37631 (local address 127.7.118.2:37839)
0929 18:02:06.657033 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:06.657111 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:06.659205 (+ 2094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:06.659999 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:06.660008 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:06.660117 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:06.660735 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:06.660751 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.661682 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.661686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:06.662996 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:06.663015 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.663817 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.663823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:06.664076 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:06.664891 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:02:06.664893 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:06.664951 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:02:06.665044 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:06.665443 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:06.665453 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:06.665459 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:02:06.665622 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 18:02:06.666006 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":121,"threads_started":1}
I20260929 18:02:06.669212 27695 heartbeater.cc:347] Connected to a master server at 127.7.118.62:37631
I20260929 18:02:06.669539 27695 heartbeater.cc:464] Registering TS with master...
I20260929 18:02:06.670361 27695 heartbeater.cc:511] Master 127.7.118.62:37631 requested a full tablet report, sending...
I20260929 18:02:06.671782 27376 ts_manager.cc:194] Registered new tserver with Master: 703db8732705428f9db94f2458144489 (127.7.118.2:34143)
I20260929 18:02:06.672732 27376 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.118.2:37839
W20260929 18:02:06.803231 27698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:06.803541 27698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:06.803586 27698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:06.808216 27698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:06.808313 27698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:06.808377 27698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.118.3
I20260929 18:02:06.814891 27698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:34661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.118.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.118.3
--webserver_port=0
--tserver_master_addrs=127.7.118.62:37631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.118.3
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:06.816088 27698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:06.817557 27698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:06.830137 27706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:06.830317 27703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:06.830765 27704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:06.830904 27698 server_base.cc:1034] running on GCE node
I20260929 18:02:06.831478 27698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:06.832293 27698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:06.833676 27698 hybrid_clock.cc:648] HybridClock initialized: now 1790704926833660 us; error 35 us; skew 500 ppm
I20260929 18:02:06.836326 27698 webserver.cc:460] Webserver started at http://127.7.118.3:45249/ using document root <none> and password file <none>
I20260929 18:02:06.837020 27698 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:06.837136 27698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:06.837397 27698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:06.839854 27698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-2/data/instance:
uuid: "f2e8023adc33418b815744fd4ba2d34d"
format_stamp: "Formatted at 2026-09-29 18:02:06 on dist-test-slave-r7ph"
server_key: "5a856cc7c7bddf9c39abad547955e3b3"
server_key_iv: "7bbefb7fa91b3b1be6579d2b6153a2a5"
server_key_version: "encryptionkey@0"
I20260929 18:02:06.840508 27698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance:
uuid: "f2e8023adc33418b815744fd4ba2d34d"
format_stamp: "Formatted at 2026-09-29 18:02:06 on dist-test-slave-r7ph"
server_key: "5a856cc7c7bddf9c39abad547955e3b3"
server_key_iv: "7bbefb7fa91b3b1be6579d2b6153a2a5"
server_key_version: "encryptionkey@0"
I20260929 18:02:06.845194 27698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:02:06.848560 27712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:06.850468 27698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:02:06.850641 27698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-2/wal
uuid: "f2e8023adc33418b815744fd4ba2d34d"
format_stamp: "Formatted at 2026-09-29 18:02:06 on dist-test-slave-r7ph"
server_key: "5a856cc7c7bddf9c39abad547955e3b3"
server_key_iv: "7bbefb7fa91b3b1be6579d2b6153a2a5"
server_key_version: "encryptionkey@0"
I20260929 18:02:06.850795 27698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:06.871098 27698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:06.871966 27698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:06.872197 27698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:06.872853 27698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:02:06.874046 27698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:02:06.874123 27698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:06.874191 27698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:02:06.874243 27698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:06.884359 27698 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.3:43885
I20260929 18:02:06.884377 27825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.3:43885 every 8 connection(s)
I20260929 18:02:06.885776 27698 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-2/data/info.pb
I20260929 18:02:06.894661 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 27698
I20260929 18:02:06.894786 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-2/wal/instance
I20260929 18:02:06.895138 7640 external_mini_cluster.cc:1428] Setting key 70af46eded97f5b61381877e537fc999
I20260929 18:02:06.897127 27415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:06.887810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.3:37649 (local address 127.7.118.62:37631)
0929 18:02:06.887952 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:06.887956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:06.889191 (+ 1235us) server_negotiation.cc:407] Connection header received
0929 18:02:06.890973 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:06.890976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:06.891030 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:06.891111 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:06.892256 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.892968 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.894488 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.894855 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.894985 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:06.896581 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:06.896585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:06.896588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:06.896604 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:06.896634 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:06.896643 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:06.896702 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:06.896903 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 18:02:06.896972 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:02:06.898161 27828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:06.888243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:37631 (local address 127.7.118.3:37649)
0929 18:02:06.888993 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:06.889044 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:06.890556 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:06.891336 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:06.891344 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:06.891444 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:06.892054 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:06.892069 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.893197 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.893201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:06.894230 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:06.894242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.895125 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.895129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:06.895384 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:06.896279 (+ 895us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:02:06.896281 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:06.896331 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:02:06.896417 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:06.896734 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:06.896746 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:06.896755 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 18:02:06.896884 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 18:02:06.897156 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":131,"threads_started":1}
I20260929 18:02:06.899514 27826 heartbeater.cc:347] Connected to a master server at 127.7.118.62:37631
I20260929 18:02:06.899876 27826 heartbeater.cc:464] Registering TS with master...
I20260929 18:02:06.900640 27826 heartbeater.cc:511] Master 127.7.118.62:37631 requested a full tablet report, sending...
I20260929 18:02:06.902148 27376 ts_manager.cc:194] Registered new tserver with Master: f2e8023adc33418b815744fd4ba2d34d (127.7.118.3:43885)
I20260929 18:02:06.902822 27376 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.118.3:37649
I20260929 18:02:06.909853 7640 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:02:06.917059 27415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:06.912279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48610 (local address 127.7.118.62:37631)
0929 18:02:06.912420 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:06.912424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:06.912491 (+ 67us) server_negotiation.cc:407] Connection header received
0929 18:02:06.912612 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:06.912615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:06.912687 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:06.912786 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:06.913180 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.913798 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.914929 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.915200 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.915299 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:06.916371 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:06.916375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:06.916382 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:06.916397 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:06.916424 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:06.916432 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:06.916517 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:06.916725 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 18:02:06.916912 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:02:06.920878 27376 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:02:06.924036 27376 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:02:06.944589 27840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:06.938797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:40203 (local address 127.0.0.1:37992)
0929 18:02:06.939452 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:06.939465 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:06.939562 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:06.940221 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:06.940224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:06.940242 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:02:06.940329 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:06.940334 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.941837 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.941842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:06.942792 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:06.942802 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.943891 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.943894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:06.944045 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:06.944093 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 18:02:06.944243 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 18:02:06.944399 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":3,"thread_start_us":99,"threads_started":1}
I20260929 18:02:06.944871 27841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:06.938812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58938 (local address 127.7.118.2:34143)
0929 18:02:06.939282 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:06.939294 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:06.939323 (+ 29us) server_negotiation.cc:407] Connection header received
0929 18:02:06.939405 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:06.939411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:06.939653 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:06.939955 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:02:06.940229 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.941646 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.943308 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.944158 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.944291 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:06.944494 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:06.944552 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 18:02:06.944673 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":94,"threads_started":1}
I20260929 18:02:06.945226 27842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:06.939096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37992 (local address 127.7.118.1:40203)
0929 18:02:06.939399 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:06.939405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:06.939567 (+ 162us) server_negotiation.cc:407] Connection header received
0929 18:02:06.939660 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:06.939667 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:06.939840 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:06.939981 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:02:06.940461 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.941731 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.942935 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.943702 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.943821 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:06.944041 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:06.944857 (+ 816us) server_negotiation.cc:299] Negotiation successful
0929 18:02:06.945041 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":84,"threads_started":1}
I20260929 18:02:06.945138 27839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:06.938553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:34143 (local address 127.0.0.1:58938)
0929 18:02:06.938952 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:06.938967 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:06.939074 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:06.939825 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:06.939829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:06.939861 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:02:06.940108 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:06.940113 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.941787 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.941790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:06.943162 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:06.943172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.944309 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.944312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:06.944399 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:06.944421 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:02:06.944485 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 18:02:06.944542 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":219,"spinlock_wait_cycles":3840,"thread_start_us":101,"threads_started":1}
I20260929 18:02:06.946573 27838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:06.937965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48600 (local address 127.7.118.3:43885)
0929 18:02:06.939259 (+ 1294us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:06.939272 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:06.939299 (+ 27us) server_negotiation.cc:407] Connection header received
0929 18:02:06.939393 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:06.939399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:06.939643 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:06.939831 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:02:06.940251 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.941407 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.945078 (+ 3671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.945896 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.946009 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:06.946184 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:06.946286 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 18:02:06.946391 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1147,"thread_start_us":101,"threads_started":1}
I20260929 18:02:06.948709 27837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:06.937898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:43885 (local address 127.0.0.1:48600)
0929 18:02:06.938525 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:06.938584 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:06.938731 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:06.939816 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:06.939822 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:06.939856 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:02:06.940091 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:06.940100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.941503 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.941510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:06.943161 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:06.943172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:06.946024 (+ 2852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:06.946028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:06.946137 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:06.946160 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:02:06.946217 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 18:02:06.946324 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":114,"spinlock_wait_cycles":2304,"thread_start_us":108,"threads_started":1}
I20260929 18:02:06.948900 27498 tablet_service.cc:1467] Processing CreateTablet for tablet 695f54e5a4a14e518173252b948d11bc (DEFAULT_TABLE table=test-table [id=cb840ca1eced4c629dfb41340586091b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:02:06.948907 27629 tablet_service.cc:1467] Processing CreateTablet for tablet 695f54e5a4a14e518173252b948d11bc (DEFAULT_TABLE table=test-table [id=cb840ca1eced4c629dfb41340586091b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:02:06.950403 27498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 695f54e5a4a14e518173252b948d11bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:06.950403 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 695f54e5a4a14e518173252b948d11bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:06.952786 27760 tablet_service.cc:1467] Processing CreateTablet for tablet 695f54e5a4a14e518173252b948d11bc (DEFAULT_TABLE table=test-table [id=cb840ca1eced4c629dfb41340586091b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:02:06.954823 27760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 695f54e5a4a14e518173252b948d11bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:06.961701 27844 tablet_bootstrap.cc:492] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489: Bootstrap starting.
I20260929 18:02:06.960029 27843 tablet_bootstrap.cc:492] T 695f54e5a4a14e518173252b948d11bc P 49af9d4ea55e4d0686ef0f831d19e699: Bootstrap starting.
I20260929 18:02:06.964356 27845 tablet_bootstrap.cc:492] T 695f54e5a4a14e518173252b948d11bc P f2e8023adc33418b815744fd4ba2d34d: Bootstrap starting.
I20260929 18:02:06.964911 27843 tablet_bootstrap.cc:654] T 695f54e5a4a14e518173252b948d11bc P 49af9d4ea55e4d0686ef0f831d19e699: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:06.964952 27844 tablet_bootstrap.cc:654] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:06.965955 27843 log.cc:824] T 695f54e5a4a14e518173252b948d11bc P 49af9d4ea55e4d0686ef0f831d19e699: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:06.966123 27844 log.cc:824] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:06.967203 27845 tablet_bootstrap.cc:654] T 695f54e5a4a14e518173252b948d11bc P f2e8023adc33418b815744fd4ba2d34d: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:06.968016 27845 log.cc:824] T 695f54e5a4a14e518173252b948d11bc P f2e8023adc33418b815744fd4ba2d34d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:06.968568 27844 tablet_bootstrap.cc:492] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489: No bootstrap required, opened a new log
I20260929 18:02:06.968568 27843 tablet_bootstrap.cc:492] T 695f54e5a4a14e518173252b948d11bc P 49af9d4ea55e4d0686ef0f831d19e699: No bootstrap required, opened a new log
I20260929 18:02:06.968803 27844 ts_tablet_manager.cc:1397] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260929 18:02:06.968802 27843 ts_tablet_manager.cc:1397] T 695f54e5a4a14e518173252b948d11bc P 49af9d4ea55e4d0686ef0f831d19e699: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260929 18:02:06.972103 27845 tablet_bootstrap.cc:492] T 695f54e5a4a14e518173252b948d11bc P f2e8023adc33418b815744fd4ba2d34d: No bootstrap required, opened a new log
I20260929 18:02:06.972376 27845 ts_tablet_manager.cc:1397] T 695f54e5a4a14e518173252b948d11bc P f2e8023adc33418b815744fd4ba2d34d: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260929 18:02:06.973883 27844 raft_consensus.cc:348] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e8023adc33418b815744fd4ba2d34d" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 43885 } } peers { permanent_uuid: "49af9d4ea55e4d0686ef0f831d19e699" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 40203 } } peers { permanent_uuid: "703db8732705428f9db94f2458144489" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34143 } }
I20260929 18:02:06.974278 27844 raft_consensus.cc:374] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:06.974339 27844 raft_consensus.cc:729] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703db8732705428f9db94f2458144489, State: Initialized, Role: FOLLOWER
I20260929 18:02:06.974386 27843 raft_consensus.cc:348] T 695f54e5a4a14e518173252b948d11bc P 49af9d4ea55e4d0686ef0f831d19e699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e8023adc33418b815744fd4ba2d34d" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 43885 } } peers { permanent_uuid: "49af9d4ea55e4d0686ef0f831d19e699" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 40203 } } peers { permanent_uuid: "703db8732705428f9db94f2458144489" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34143 } }
I20260929 18:02:06.974843 27843 raft_consensus.cc:374] T 695f54e5a4a14e518173252b948d11bc P 49af9d4ea55e4d0686ef0f831d19e699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:06.974913 27843 raft_consensus.cc:729] T 695f54e5a4a14e518173252b948d11bc P 49af9d4ea55e4d0686ef0f831d19e699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49af9d4ea55e4d0686ef0f831d19e699, State: Initialized, Role: FOLLOWER
I20260929 18:02:06.975065 27844 consensus_queue.cc:260] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e8023adc33418b815744fd4ba2d34d" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 43885 } } peers { permanent_uuid: "49af9d4ea55e4d0686ef0f831d19e699" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 40203 } } peers { permanent_uuid: "703db8732705428f9db94f2458144489" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34143 } }
I20260929 18:02:06.975509 27843 consensus_queue.cc:260] T 695f54e5a4a14e518173252b948d11bc P 49af9d4ea55e4d0686ef0f831d19e699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e8023adc33418b815744fd4ba2d34d" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 43885 } } peers { permanent_uuid: "49af9d4ea55e4d0686ef0f831d19e699" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 40203 } } peers { permanent_uuid: "703db8732705428f9db94f2458144489" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34143 } }
I20260929 18:02:06.976413 27695 heartbeater.cc:503] Master 127.7.118.62:37631 was elected leader, sending a full tablet report...
I20260929 18:02:06.976913 27844 ts_tablet_manager.cc:1428] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260929 18:02:06.976991 27843 ts_tablet_manager.cc:1428] T 695f54e5a4a14e518173252b948d11bc P 49af9d4ea55e4d0686ef0f831d19e699: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260929 18:02:06.977705 27564 heartbeater.cc:503] Master 127.7.118.62:37631 was elected leader, sending a full tablet report...
I20260929 18:02:06.979478 27845 raft_consensus.cc:348] T 695f54e5a4a14e518173252b948d11bc P f2e8023adc33418b815744fd4ba2d34d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e8023adc33418b815744fd4ba2d34d" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 43885 } } peers { permanent_uuid: "49af9d4ea55e4d0686ef0f831d19e699" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 40203 } } peers { permanent_uuid: "703db8732705428f9db94f2458144489" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34143 } }
I20260929 18:02:06.979986 27845 raft_consensus.cc:374] T 695f54e5a4a14e518173252b948d11bc P f2e8023adc33418b815744fd4ba2d34d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:06.980059 27845 raft_consensus.cc:729] T 695f54e5a4a14e518173252b948d11bc P f2e8023adc33418b815744fd4ba2d34d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2e8023adc33418b815744fd4ba2d34d, State: Initialized, Role: FOLLOWER
I20260929 18:02:06.980602 27845 consensus_queue.cc:260] T 695f54e5a4a14e518173252b948d11bc P f2e8023adc33418b815744fd4ba2d34d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e8023adc33418b815744fd4ba2d34d" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 43885 } } peers { permanent_uuid: "49af9d4ea55e4d0686ef0f831d19e699" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 40203 } } peers { permanent_uuid: "703db8732705428f9db94f2458144489" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34143 } }
I20260929 18:02:06.981755 27826 heartbeater.cc:503] Master 127.7.118.62:37631 was elected leader, sending a full tablet report...
I20260929 18:02:06.981832 27845 ts_tablet_manager.cc:1428] T 695f54e5a4a14e518173252b948d11bc P f2e8023adc33418b815744fd4ba2d34d: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260929 18:02:07.029778 27849 raft_consensus.cc:482] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:02:07.030121 27849 raft_consensus.cc:504] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e8023adc33418b815744fd4ba2d34d" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 43885 } } peers { permanent_uuid: "49af9d4ea55e4d0686ef0f831d19e699" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 40203 } } peers { permanent_uuid: "703db8732705428f9db94f2458144489" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34143 } }
I20260929 18:02:07.031463 27849 leader_election.cc:288] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2e8023adc33418b815744fd4ba2d34d (127.7.118.3:43885), 49af9d4ea55e4d0686ef0f831d19e699 (127.7.118.1:40203)
I20260929 18:02:07.036554 27838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:07.032268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:48029 (local address 127.7.118.3:43885)
0929 18:02:07.032416 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:07.032423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:07.032438 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:02:07.032498 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:07.032502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:07.032548 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:07.032640 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:02:07.033151 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:07.033807 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:07.035406 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:07.035852 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:07.035959 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:07.036025 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:07.036315 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 18:02:07.036412 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 18:02:07.036545 27697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:07.031944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.3:43885 (local address 127.7.118.2:48029)
0929 18:02:07.032207 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:07.032224 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:07.032349 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:07.032692 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:07.032694 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:07.032717 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:02:07.032951 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:07.032959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:07.033933 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:07.033937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:07.035253 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:07.035265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:07.036010 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:07.036014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:07.036149 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:07.036178 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 18:02:07.036263 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 18:02:07.036365 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260929 18:02:07.036746 27842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:07.032362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.118.2:42677 (local address 127.7.118.1:40203)
0929 18:02:07.032492 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:07.032497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:07.032873 (+ 376us) server_negotiation.cc:407] Connection header received
0929 18:02:07.032945 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:07.032948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:07.033060 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:07.033158 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:02:07.033532 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:07.034199 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:07.035513 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:07.036001 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:07.036168 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:07.036249 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:07.036499 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 18:02:07.036628 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:02:07.036759 27852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:07.032236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:40203 (local address 127.7.118.2:42677)
0929 18:02:07.032674 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:07.032688 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:07.032805 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:07.033184 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:07.033187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:07.033207 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:02:07.033334 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:07.033341 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:07.034568 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:07.034571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:07.035320 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:07.035359 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:07.036204 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:07.036207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:07.036355 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:07.036379 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:02:07.036463 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 18:02:07.036532 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":156,"threads_started":1}
I20260929 18:02:07.037446 27518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "695f54e5a4a14e518173252b948d11bc" candidate_uuid: "703db8732705428f9db94f2458144489" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49af9d4ea55e4d0686ef0f831d19e699" is_pre_election: true
I20260929 18:02:07.037448 27780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "695f54e5a4a14e518173252b948d11bc" candidate_uuid: "703db8732705428f9db94f2458144489" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2e8023adc33418b815744fd4ba2d34d" is_pre_election: true
I20260929 18:02:07.037796 27518 raft_consensus.cc:2454] T 695f54e5a4a14e518173252b948d11bc P 49af9d4ea55e4d0686ef0f831d19e699 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703db8732705428f9db94f2458144489 in term 0.
I20260929 18:02:07.037811 27780 raft_consensus.cc:2454] T 695f54e5a4a14e518173252b948d11bc P f2e8023adc33418b815744fd4ba2d34d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703db8732705428f9db94f2458144489 in term 0.
I20260929 18:02:07.038471 27582 leader_election.cc:302] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 703db8732705428f9db94f2458144489, f2e8023adc33418b815744fd4ba2d34d; no voters:
I20260929 18:02:07.038883 27849 raft_consensus.cc:2781] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:02:07.039126 27849 raft_consensus.cc:482] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:02:07.039237 27849 raft_consensus.cc:3037] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:07.040752 27849 raft_consensus.cc:504] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e8023adc33418b815744fd4ba2d34d" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 43885 } } peers { permanent_uuid: "49af9d4ea55e4d0686ef0f831d19e699" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 40203 } } peers { permanent_uuid: "703db8732705428f9db94f2458144489" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34143 } }
I20260929 18:02:07.041524 27849 leader_election.cc:288] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489 [CANDIDATE]: Term 1 election: Requested vote from peers f2e8023adc33418b815744fd4ba2d34d (127.7.118.3:43885), 49af9d4ea55e4d0686ef0f831d19e699 (127.7.118.1:40203)
I20260929 18:02:07.042215 27780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "695f54e5a4a14e518173252b948d11bc" candidate_uuid: "703db8732705428f9db94f2458144489" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2e8023adc33418b815744fd4ba2d34d"
I20260929 18:02:07.042331 27518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "695f54e5a4a14e518173252b948d11bc" candidate_uuid: "703db8732705428f9db94f2458144489" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49af9d4ea55e4d0686ef0f831d19e699"
I20260929 18:02:07.042416 27780 raft_consensus.cc:3037] T 695f54e5a4a14e518173252b948d11bc P f2e8023adc33418b815744fd4ba2d34d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:07.042529 27518 raft_consensus.cc:3037] T 695f54e5a4a14e518173252b948d11bc P 49af9d4ea55e4d0686ef0f831d19e699 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:07.043742 27518 raft_consensus.cc:2454] T 695f54e5a4a14e518173252b948d11bc P 49af9d4ea55e4d0686ef0f831d19e699 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703db8732705428f9db94f2458144489 in term 1.
I20260929 18:02:07.043743 27780 raft_consensus.cc:2454] T 695f54e5a4a14e518173252b948d11bc P f2e8023adc33418b815744fd4ba2d34d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703db8732705428f9db94f2458144489 in term 1.
I20260929 18:02:07.044193 27582 leader_election.cc:302] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49af9d4ea55e4d0686ef0f831d19e699, 703db8732705428f9db94f2458144489; no voters:
I20260929 18:02:07.044423 27849 raft_consensus.cc:2781] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:07.044700 27849 raft_consensus.cc:686] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489 [term 1 LEADER]: Becoming Leader. State: Replica: 703db8732705428f9db94f2458144489, State: Running, Role: LEADER
I20260929 18:02:07.045324 27849 consensus_queue.cc:237] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e8023adc33418b815744fd4ba2d34d" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 43885 } } peers { permanent_uuid: "49af9d4ea55e4d0686ef0f831d19e699" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 40203 } } peers { permanent_uuid: "703db8732705428f9db94f2458144489" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34143 } }
I20260929 18:02:07.049557 27376 catalog_manager.cc:5495] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489 reported cstate change: term changed from 0 to 1, leader changed from <none> to 703db8732705428f9db94f2458144489 (127.7.118.2). New cstate: current_term: 1 leader_uuid: "703db8732705428f9db94f2458144489" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e8023adc33418b815744fd4ba2d34d" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 43885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49af9d4ea55e4d0686ef0f831d19e699" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 40203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "703db8732705428f9db94f2458144489" member_type: VOTER last_known_addr { host: "127.7.118.2" port: 34143 } health_report { overall_health: HEALTHY } } }
I20260929 18:02:07.075237 27841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:07.070806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58952 (local address 127.7.118.2:34143)
0929 18:02:07.071178 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:07.071183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:07.071198 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:02:07.071292 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:07.071294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:07.071365 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:07.071468 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:02:07.071816 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:07.072372 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:07.073375 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:07.073617 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:07.073719 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:07.074134 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:02:07.074655 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:02:07.074768 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:07.074905 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 18:02:07.074964 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237}
I20260929 18:02:07.082228 27518 raft_consensus.cc:1261] T 695f54e5a4a14e518173252b948d11bc P 49af9d4ea55e4d0686ef0f831d19e699 [term 1 FOLLOWER]: Refusing update from remote peer 703db8732705428f9db94f2458144489: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:02:07.082378 27780 raft_consensus.cc:1261] T 695f54e5a4a14e518173252b948d11bc P f2e8023adc33418b815744fd4ba2d34d [term 1 FOLLOWER]: Refusing update from remote peer 703db8732705428f9db94f2458144489: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:02:07.083467 27849 consensus_queue.cc:1035] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2e8023adc33418b815744fd4ba2d34d" member_type: VOTER last_known_addr { host: "127.7.118.3" port: 43885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:02:07.083915 27853 consensus_queue.cc:1035] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49af9d4ea55e4d0686ef0f831d19e699" member_type: VOTER last_known_addr { host: "127.7.118.1" port: 40203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:02:07.093978 27858 mvcc.cc:205] Tried to move back new op lower bound from 7334727381319417856 to 7334727381182509056. Current Snapshot: MvccSnapshot[applied={T|T < 7334727381319417856}]
I20260929 18:02:07.094679 27857 mvcc.cc:205] Tried to move back new op lower bound from 7334727381319417856 to 7334727381182509056. Current Snapshot: MvccSnapshot[applied={T|T < 7334727381319417856}]
I20260929 18:02:07.106686 27376 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:02:07.106869 27376 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:02:07.110626 27376 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e90f41d100234b8e886eba7281c4d094: Sending DeleteTablet for 3 replicas of tablet 695f54e5a4a14e518173252b948d11bc
I20260929 18:02:07.111881 27760 tablet_service.cc:1514] Processing DeleteTablet for tablet 695f54e5a4a14e518173252b948d11bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:02:07 UTC) from {username='slave'} at 127.0.0.1:48600
I20260929 18:02:07.111879 27629 tablet_service.cc:1514] Processing DeleteTablet for tablet 695f54e5a4a14e518173252b948d11bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:02:07 UTC) from {username='slave'} at 127.0.0.1:58938
I20260929 18:02:07.112114 27498 tablet_service.cc:1514] Processing DeleteTablet for tablet 695f54e5a4a14e518173252b948d11bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:02:07 UTC) from {username='slave'} at 127.0.0.1:37992
I20260929 18:02:07.112604 27869 tablet_replica.cc:330] T 695f54e5a4a14e518173252b948d11bc P f2e8023adc33418b815744fd4ba2d34d: stopping tablet replica
I20260929 18:02:07.112653 27870 tablet_replica.cc:330] T 695f54e5a4a14e518173252b948d11bc P 49af9d4ea55e4d0686ef0f831d19e699: stopping tablet replica
I20260929 18:02:07.112996 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 27436
I20260929 18:02:07.113044 27869 raft_consensus.cc:2229] T 695f54e5a4a14e518173252b948d11bc P f2e8023adc33418b815744fd4ba2d34d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:02:07.115437 27869 raft_consensus.cc:2258] T 695f54e5a4a14e518173252b948d11bc P f2e8023adc33418b815744fd4ba2d34d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:02:07.115474 27868 tablet_replica.cc:330] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489: stopping tablet replica
I20260929 18:02:07.115846 27868 raft_consensus.cc:2229] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:02:07.116813 27868 raft_consensus.cc:2258] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:02:07.117451 27869 ts_tablet_manager.cc:1905] T 695f54e5a4a14e518173252b948d11bc P f2e8023adc33418b815744fd4ba2d34d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:02:07.118721 27868 ts_tablet_manager.cc:1905] T 695f54e5a4a14e518173252b948d11bc P 703db8732705428f9db94f2458144489: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 18:02:07.121307 27360 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.118.1:40203 (error 108)
I20260929 18:02:07.121587 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 27567
I20260929 18:02:07.122449 27869 ts_tablet_manager.cc:1918] T 695f54e5a4a14e518173252b948d11bc P f2e8023adc33418b815744fd4ba2d34d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:02:07.122696 27869 log.cc:1196] T 695f54e5a4a14e518173252b948d11bc P f2e8023adc33418b815744fd4ba2d34d: Deleting WAL directory at /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704657566865-7640-0/minicluster-data/ts-2/wal/wals/695f54e5a4a14e518173252b948d11bc
I20260929 18:02:07.123160 27869 ts_tablet_manager.cc:1939] T 695f54e5a4a14e518173252b948d11bc P f2e8023adc33418b815744fd4ba2d34d: Deleting consensus metadata
I20260929 18:02:07.124307 27837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:07.123899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.1:40203 (local address 127.0.0.1:37998)
0929 18:02:07.124077 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:07.124195 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.1:40203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 18:02:07.124570 27360 catalog_manager.cc:4584] TS 49af9d4ea55e4d0686ef0f831d19e699 (127.7.118.1:40203): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 695f54e5a4a14e518173252b948d11bc: Network error: Client connection negotiation failed: client connection to 127.7.118.1:40203: connect: Connection refused (error 111)
I20260929 18:02:07.124871 27360 catalog_manager.cc:4841] TS f2e8023adc33418b815744fd4ba2d34d (127.7.118.3:43885): tablet 695f54e5a4a14e518173252b948d11bc (table test-table [id=cb840ca1eced4c629dfb41340586091b]) successfully deleted
I20260929 18:02:07.128907 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 27698
I20260929 18:02:07.130191 27837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:07.129734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.2:34143 (local address 127.0.0.1:58958)
0929 18:02:07.130009 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:07.130079 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.2:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260929 18:02:07.136816 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 27345
2026-09-29T18:02:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1246 ms)
[----------] 6 tests from EncryptionPolicyTest (11333 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:02:07 dist-test-slave-r7ph krb5kdc[27874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:02:07 dist-test-slave-r7ph krb5kdc[27874](info): set up 2 sockets
Sep 29 18:02:07 dist-test-slave-r7ph krb5kdc[27874](info): commencing operation
krb5kdc: starting...
W20260929 18:02:08.222878 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 18:02:08 dist-test-slave-r7ph krb5kdc[27874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:02:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:02:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:02:08.378918 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34069
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:08.498535 27887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:08.498909 27887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:08.499001 27887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:08.503279 27887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:08.503398 27887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:08.503438 27887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:08.503485 27887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:08.503527 27887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:08.509202 27887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34069
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:08.510469 27887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:08.511622 27887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:02:08.522311 27887 server_base.cc:1034] running on GCE node
W20260929 18:02:08.522337 27892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:08.522297 27895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:08.522337 27893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:08.523281 27887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:08.524653 27887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:08.525857 27887 hybrid_clock.cc:648] HybridClock initialized: now 1790704928525840 us; error 35 us; skew 500 ppm
Sep 29 18:02:08 dist-test-slave-r7ph krb5kdc[27874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:08.529209 27887 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:02:08.530411 27887 webserver.cc:460] Webserver started at http://127.0.0.1:40033/ using document root <none> and password file <none>
I20260929 18:02:08.531035 27887 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:08.531131 27887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:08.531370 27887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:08.533540 27887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "d7f0daafe2954d78a4666ad152528ac0"
format_stamp: "Formatted at 2026-09-29 18:02:08 on dist-test-slave-r7ph"
server_key: "88aa472273db270d4d17ff8f695cf9ec"
server_key_iv: "016d1c10e7d5330aeb3f6358e46c46b9"
server_key_version: "encryptionkey@0"
I20260929 18:02:08.534159 27887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "d7f0daafe2954d78a4666ad152528ac0"
format_stamp: "Formatted at 2026-09-29 18:02:08 on dist-test-slave-r7ph"
server_key: "88aa472273db270d4d17ff8f695cf9ec"
server_key_iv: "016d1c10e7d5330aeb3f6358e46c46b9"
server_key_version: "encryptionkey@0"
I20260929 18:02:08.538187 27887 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260929 18:02:08.540916 27902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:08.542755 27887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:02:08.542918 27887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "d7f0daafe2954d78a4666ad152528ac0"
format_stamp: "Formatted at 2026-09-29 18:02:08 on dist-test-slave-r7ph"
server_key: "88aa472273db270d4d17ff8f695cf9ec"
server_key_iv: "016d1c10e7d5330aeb3f6358e46c46b9"
server_key_version: "encryptionkey@0"
I20260929 18:02:08.543071 27887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:08.556890 27887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:08.560858 27887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:08.561146 27887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:08.568974 27887 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34069
I20260929 18:02:08.569000 27954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34069 every 8 connection(s)
I20260929 18:02:08.570099 27887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:02:08.573202 27955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:08.576458 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 27887
I20260929 18:02:08.576634 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:02:08.576958 7640 external_mini_cluster.cc:1428] Setting key a2806d0859f10d27673dd5a54376d3c6
I20260929 18:02:08.580099 27955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7f0daafe2954d78a4666ad152528ac0: Bootstrap starting.
I20260929 18:02:08.582548 27955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7f0daafe2954d78a4666ad152528ac0: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:08.583384 27955 log.cc:824] T 00000000000000000000000000000000 P d7f0daafe2954d78a4666ad152528ac0: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:08.586025 27955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7f0daafe2954d78a4666ad152528ac0: No bootstrap required, opened a new log
Sep 29 18:02:08 dist-test-slave-r7ph krb5kdc[27874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:02:08.591192 27955 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7f0daafe2954d78a4666ad152528ac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f0daafe2954d78a4666ad152528ac0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34069 } }
I20260929 18:02:08.591531 27955 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7f0daafe2954d78a4666ad152528ac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:08.591584 27955 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7f0daafe2954d78a4666ad152528ac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f0daafe2954d78a4666ad152528ac0, State: Initialized, Role: FOLLOWER
I20260929 18:02:08.592139 27955 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7f0daafe2954d78a4666ad152528ac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f0daafe2954d78a4666ad152528ac0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34069 } }
I20260929 18:02:08.592310 27955 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7f0daafe2954d78a4666ad152528ac0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:02:08.592379 27955 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7f0daafe2954d78a4666ad152528ac0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:02:08.592475 27955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7f0daafe2954d78a4666ad152528ac0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:08.593781 27955 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7f0daafe2954d78a4666ad152528ac0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f0daafe2954d78a4666ad152528ac0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34069 } }
I20260929 18:02:08.594205 27955 leader_election.cc:302] T 00000000000000000000000000000000 P d7f0daafe2954d78a4666ad152528ac0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7f0daafe2954d78a4666ad152528ac0; no voters:
I20260929 18:02:08.594604 27958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:08.578729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54141 (local address 127.0.0.1:34069)
0929 18:02:08.579297 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:08.579313 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:08.579355 (+ 42us) server_negotiation.cc:407] Connection header received
0929 18:02:08.580036 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:08.580055 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:08.580503 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:08.580919 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:08.581840 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:08.583019 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:08.584080 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:08.584822 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:08.584989 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:08.588279 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:08.588286 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:08.588308 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:08.588346 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:08.590725 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:08.591392 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:08.591399 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:08.591405 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:08.591576 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:08.592012 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:08.592015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:08.592017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:08.592597 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:08.592843 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:08.593152 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 18:02:08.593404 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":161,"threads_started":1}
I20260929 18:02:08.594878 27960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7f0daafe2954d78a4666ad152528ac0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:08.594614 27955 leader_election.cc:288] T 00000000000000000000000000000000 P d7f0daafe2954d78a4666ad152528ac0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:02:08.595245 27960 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7f0daafe2954d78a4666ad152528ac0 [term 1 LEADER]: Becoming Leader. State: Replica: d7f0daafe2954d78a4666ad152528ac0, State: Running, Role: LEADER
I20260929 18:02:08.595628 27960 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7f0daafe2954d78a4666ad152528ac0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f0daafe2954d78a4666ad152528ac0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34069 } }
I20260929 18:02:08.596374 27955 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7f0daafe2954d78a4666ad152528ac0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:02:08.597540 27961 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7f0daafe2954d78a4666ad152528ac0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7f0daafe2954d78a4666ad152528ac0. Latest consensus state: current_term: 1 leader_uuid: "d7f0daafe2954d78a4666ad152528ac0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f0daafe2954d78a4666ad152528ac0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34069 } } }
I20260929 18:02:08.597540 27962 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7f0daafe2954d78a4666ad152528ac0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7f0daafe2954d78a4666ad152528ac0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f0daafe2954d78a4666ad152528ac0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34069 } } }
I20260929 18:02:08.597705 27961 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7f0daafe2954d78a4666ad152528ac0 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:08.597705 27962 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7f0daafe2954d78a4666ad152528ac0 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:08.598946 7640 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260929 18:02:08.602730 27975 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d7f0daafe2954d78a4666ad152528ac0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:02:08.602918 27975 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:02:08.603053 27969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:02:08.603969 27969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:02:08.611899 27969 catalog_manager.cc:1347] Generated new cluster ID: f486dd5165c34f62924ba616ffef7ce5
I20260929 18:02:08.611984 27969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:02:08.612469 27958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:08.601582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40997 (local address 127.0.0.1:34069)
0929 18:02:08.601688 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:08.601692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:08.601707 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:02:08.601750 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:08.601752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:08.601797 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:08.601878 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:08.602945 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:08.603742 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:08.605019 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:08.605403 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:08.605528 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:08.607700 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:08.607705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:08.607708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:08.607743 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:08.609928 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:08.610705 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:08.610709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:08.610711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:08.610816 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:08.611215 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:08.611221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:08.611224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:08.611463 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:08.611629 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:08.612187 (+ 558us) server_negotiation.cc:299] Negotiation successful
0929 18:02:08.612307 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:02:08.627565 27969 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:02:08.628690 27969 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:02:08.638160 27969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7f0daafe2954d78a4666ad152528ac0: Generated new TSK 0
I20260929 18:02:08.639048 27969 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:02:08.645458 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 27887
W20260929 18:02:08.651507 27872 connection.cc:535] client connection to 127.0.0.1:34069 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34069): Connection reset by peer (error 104)
2026-09-29T18:02:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1507 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:02:08 dist-test-slave-r7ph krb5kdc[27986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:02:08 dist-test-slave-r7ph krb5kdc[27986](info): set up 2 sockets
Sep 29 18:02:08 dist-test-slave-r7ph krb5kdc[27986](info): commencing operation
krb5kdc: starting...
W20260929 18:02:09.754591 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 18:02:09 dist-test-slave-r7ph krb5kdc[27986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:02:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:02:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:02:09.890652 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43499
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:10.009826 27999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:10.010200 27999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:10.010282 27999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:10.014626 27999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:10.014711 27999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:10.014732 27999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:10.014750 27999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:10.014765 27999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:10.020623 27999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43499
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:10.021770 27999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:10.022845 27999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:10.032271 28004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:10.032256 28007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:10.032608 27999 server_base.cc:1034] running on GCE node
W20260929 18:02:10.032259 28005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:10.033408 27999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:10.034757 27999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:10.035989 27999 hybrid_clock.cc:648] HybridClock initialized: now 1790704930035942 us; error 63 us; skew 500 ppm
Sep 29 18:02:10 dist-test-slave-r7ph krb5kdc[27986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:10.039326 27999 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:02:10.040442 27999 webserver.cc:460] Webserver started at http://127.0.0.1:38315/ using document root <none> and password file <none>
I20260929 18:02:10.041082 27999 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:10.041184 27999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:10.041437 27999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:10.043793 27999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "140993dfa668452c821d1838cdf52c5c"
format_stamp: "Formatted at 2026-09-29 18:02:10 on dist-test-slave-r7ph"
server_key: "212f9aef72a712a5c0b2b1106b2a4a92"
server_key_iv: "24e76f8490036fff05b04edf9fc74421"
server_key_version: "encryptionkey@0"
I20260929 18:02:10.044413 27999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "140993dfa668452c821d1838cdf52c5c"
format_stamp: "Formatted at 2026-09-29 18:02:10 on dist-test-slave-r7ph"
server_key: "212f9aef72a712a5c0b2b1106b2a4a92"
server_key_iv: "24e76f8490036fff05b04edf9fc74421"
server_key_version: "encryptionkey@0"
I20260929 18:02:10.048822 27999 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:02:10.051888 28014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:10.053323 27999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 18:02:10.053494 27999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "140993dfa668452c821d1838cdf52c5c"
format_stamp: "Formatted at 2026-09-29 18:02:10 on dist-test-slave-r7ph"
server_key: "212f9aef72a712a5c0b2b1106b2a4a92"
server_key_iv: "24e76f8490036fff05b04edf9fc74421"
server_key_version: "encryptionkey@0"
I20260929 18:02:10.053645 27999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:10.067879 27999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:10.071655 27999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:10.071893 27999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:10.079770 27999 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43499
I20260929 18:02:10.079803 28066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43499 every 8 connection(s)
I20260929 18:02:10.080875 27999 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:02:10.083840 28067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:10.087765 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 27999
I20260929 18:02:10.087996 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:02:10.088315 7640 external_mini_cluster.cc:1428] Setting key 0b05b0c5588d388fea989b3a410060b8
I20260929 18:02:10.090528 28067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 140993dfa668452c821d1838cdf52c5c: Bootstrap starting.
I20260929 18:02:10.093534 28067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 140993dfa668452c821d1838cdf52c5c: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:10.094512 28067 log.cc:824] T 00000000000000000000000000000000 P 140993dfa668452c821d1838cdf52c5c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:10.097101 28067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 140993dfa668452c821d1838cdf52c5c: No bootstrap required, opened a new log
I20260929 18:02:10.100136 28070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:10.089918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44095 (local address 127.0.0.1:43499)
0929 18:02:10.090436 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:10.090457 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:10.090502 (+ 45us) server_negotiation.cc:407] Connection header received
0929 18:02:10.091387 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:10.091411 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:10.091906 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:10.092325 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:10.092767 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:10.094323 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:10.095670 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:10.096376 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:10.096561 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:10.097599 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:10.097605 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:10.097623 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:10.098154 (+ 531us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:10.098231 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:10.098294 (+ 63us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:10.098436 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:10.098670 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 18:02:10.098821 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":146,"threads_started":1}
I20260929 18:02:10.101243 28067 raft_consensus.cc:348] T 00000000000000000000000000000000 P 140993dfa668452c821d1838cdf52c5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "140993dfa668452c821d1838cdf52c5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43499 } }
I20260929 18:02:10.101502 28067 raft_consensus.cc:374] T 00000000000000000000000000000000 P 140993dfa668452c821d1838cdf52c5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:10.101590 28067 raft_consensus.cc:729] T 00000000000000000000000000000000 P 140993dfa668452c821d1838cdf52c5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 140993dfa668452c821d1838cdf52c5c, State: Initialized, Role: FOLLOWER
I20260929 18:02:10.102114 28067 consensus_queue.cc:260] T 00000000000000000000000000000000 P 140993dfa668452c821d1838cdf52c5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "140993dfa668452c821d1838cdf52c5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43499 } }
I20260929 18:02:10.102304 28067 raft_consensus.cc:388] T 00000000000000000000000000000000 P 140993dfa668452c821d1838cdf52c5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:02:10.102391 28067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 140993dfa668452c821d1838cdf52c5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:02:10.102531 28067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 140993dfa668452c821d1838cdf52c5c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:10.103593 28067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 140993dfa668452c821d1838cdf52c5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "140993dfa668452c821d1838cdf52c5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43499 } }
I20260929 18:02:10.103874 7640 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:02:10.104035 28067 leader_election.cc:302] T 00000000000000000000000000000000 P 140993dfa668452c821d1838cdf52c5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 140993dfa668452c821d1838cdf52c5c; no voters:
I20260929 18:02:10.104422 28067 leader_election.cc:288] T 00000000000000000000000000000000 P 140993dfa668452c821d1838cdf52c5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:02:10.104524 28072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 140993dfa668452c821d1838cdf52c5c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:10.104815 28072 raft_consensus.cc:686] T 00000000000000000000000000000000 P 140993dfa668452c821d1838cdf52c5c [term 1 LEADER]: Becoming Leader. State: Replica: 140993dfa668452c821d1838cdf52c5c, State: Running, Role: LEADER
I20260929 18:02:10.105280 28072 consensus_queue.cc:237] T 00000000000000000000000000000000 P 140993dfa668452c821d1838cdf52c5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "140993dfa668452c821d1838cdf52c5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43499 } }
I20260929 18:02:10.106021 28067 sys_catalog.cc:564] T 00000000000000000000000000000000 P 140993dfa668452c821d1838cdf52c5c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:02:10.107466 28074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 140993dfa668452c821d1838cdf52c5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 140993dfa668452c821d1838cdf52c5c. Latest consensus state: current_term: 1 leader_uuid: "140993dfa668452c821d1838cdf52c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "140993dfa668452c821d1838cdf52c5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43499 } } }
I20260929 18:02:10.107612 28074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 140993dfa668452c821d1838cdf52c5c [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:10.107978 28073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 140993dfa668452c821d1838cdf52c5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "140993dfa668452c821d1838cdf52c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "140993dfa668452c821d1838cdf52c5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43499 } } }
I20260929 18:02:10.108065 28073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 140993dfa668452c821d1838cdf52c5c [sys.catalog]: This master's current role is: LEADER
W20260929 18:02:10.111586 28092 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 140993dfa668452c821d1838cdf52c5c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:02:10.111694 28092 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:02:10.111781 28086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:02:10.112206 28070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:10.106373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42525 (local address 127.0.0.1:43499)
0929 18:02:10.106495 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:10.106500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:10.107114 (+ 614us) server_negotiation.cc:407] Connection header received
0929 18:02:10.107168 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:10.107173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:10.107236 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:10.107325 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:10.107801 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:10.108734 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:10.109927 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:10.110352 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:10.110536 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:10.111575 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:10.111580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:10.111583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:10.111600 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:10.111634 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:10.111650 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:10.111756 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:10.111992 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 18:02:10.112029 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:02:10.112867 28086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:02:10.119658 28086 catalog_manager.cc:1347] Generated new cluster ID: dfcd8473c2b84e759925555a9b6b071d
I20260929 18:02:10.119776 28086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:02:10.129876 28086 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:02:10.131011 28086 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:02:10.137301 28086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 140993dfa668452c821d1838cdf52c5c: Generated new TSK 0
I20260929 18:02:10.138322 28086 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:02:10.147290 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 27999
W20260929 18:02:10.153010 27984 connection.cc:535] client connection to 127.0.0.1:43499 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43499): Connection reset by peer (error 104)
2026-09-29T18:02:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1501 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:02:10 dist-test-slave-r7ph krb5kdc[28098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:02:10 dist-test-slave-r7ph krb5kdc[28098](info): set up 2 sockets
Sep 29 18:02:10 dist-test-slave-r7ph krb5kdc[28098](info): commencing operation
krb5kdc: starting...
W20260929 18:02:11.202692 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 18:02:11 dist-test-slave-r7ph krb5kdc[28098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:02:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:02:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:02:11.342545 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33691
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:11.464449 28111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:11.464775 28111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:11.464849 28111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:11.469899 28111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:11.469996 28111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:11.470018 28111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:11.470038 28111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:11.470054 28111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:11.476033 28111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33691
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:11.477267 28111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:11.478425 28111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:11.488230 28119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:11.488230 28117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:11.488230 28116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:11.488687 28111 server_base.cc:1034] running on GCE node
I20260929 18:02:11.489652 28111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:11.491119 28111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:11.492329 28111 hybrid_clock.cc:648] HybridClock initialized: now 1790704931492292 us; error 56 us; skew 500 ppm
Sep 29 18:02:11 dist-test-slave-r7ph krb5kdc[28098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:11.504360 28111 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:02:11.505548 28111 webserver.cc:460] Webserver started at http://127.0.0.1:44333/ using document root <none> and password file <none>
I20260929 18:02:11.506213 28111 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:11.506366 28111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:11.506655 28111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:11.508980 28111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "3c7fe7c8d737433289a65cff21df0414"
format_stamp: "Formatted at 2026-09-29 18:02:11 on dist-test-slave-r7ph"
server_key: "2fb9ab3bae6897018785a080413161d4"
server_key_iv: "9b8d5feeb30b921681bf6018c5b82432"
server_key_version: "encryptionkey@0"
I20260929 18:02:11.509706 28111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "3c7fe7c8d737433289a65cff21df0414"
format_stamp: "Formatted at 2026-09-29 18:02:11 on dist-test-slave-r7ph"
server_key: "2fb9ab3bae6897018785a080413161d4"
server_key_iv: "9b8d5feeb30b921681bf6018c5b82432"
server_key_version: "encryptionkey@0"
I20260929 18:02:11.514209 28111 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260929 18:02:11.517202 28126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:11.518764 28111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:02:11.518934 28111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "3c7fe7c8d737433289a65cff21df0414"
format_stamp: "Formatted at 2026-09-29 18:02:11 on dist-test-slave-r7ph"
server_key: "2fb9ab3bae6897018785a080413161d4"
server_key_iv: "9b8d5feeb30b921681bf6018c5b82432"
server_key_version: "encryptionkey@0"
I20260929 18:02:11.519081 28111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:11.524101 28111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:11.527616 28111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:11.527786 28111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:11.536551 28111 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33691
I20260929 18:02:11.536584 28178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33691 every 8 connection(s)
I20260929 18:02:11.537622 28111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:02:11.539769 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 28111
I20260929 18:02:11.539970 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:02:11.540297 7640 external_mini_cluster.cc:1428] Setting key 059381118442bd2badaf8aaa6b1b4bfe
I20260929 18:02:11.540977 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:11.546752 28182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:11.541844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55557 (local address 127.0.0.1:33691)
0929 18:02:11.542512 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:11.542536 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:11.542701 (+ 165us) server_negotiation.cc:407] Connection header received
0929 18:02:11.543609 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:11.543629 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:11.543989 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:11.544391 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:11.545265 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:11.545270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:11.545285 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:11.545297 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:11.545360 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:11.545380 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:11.545556 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:11.545633 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 18:02:11.545816 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":173,"threads_started":1}
I20260929 18:02:11.549046 28179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c7fe7c8d737433289a65cff21df0414: Bootstrap starting.
I20260929 18:02:11.550340 7640 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:02:11.551877 28179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c7fe7c8d737433289a65cff21df0414: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:11.552748 28179 log.cc:824] T 00000000000000000000000000000000 P 3c7fe7c8d737433289a65cff21df0414: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:11.554744 28182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:11.552628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36943 (local address 127.0.0.1:33691)
0929 18:02:11.552781 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:11.552788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:11.553070 (+ 282us) server_negotiation.cc:407] Connection header received
0929 18:02:11.553252 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:11.553255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:11.553290 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:11.553364 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:11.554294 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:11.554298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:11.554301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:11.554314 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:11.554345 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:11.554358 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:11.554455 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:11.554564 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 18:02:11.554608 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 18:02:11.555510 28179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c7fe7c8d737433289a65cff21df0414: No bootstrap required, opened a new log
I20260929 18:02:11.559333 28179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c7fe7c8d737433289a65cff21df0414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7fe7c8d737433289a65cff21df0414" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33691 } }
I20260929 18:02:11.559612 28179 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c7fe7c8d737433289a65cff21df0414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:11.559648 28179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c7fe7c8d737433289a65cff21df0414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c7fe7c8d737433289a65cff21df0414, State: Initialized, Role: FOLLOWER
I20260929 18:02:11.560111 28179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c7fe7c8d737433289a65cff21df0414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7fe7c8d737433289a65cff21df0414" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33691 } }
I20260929 18:02:11.560240 28179 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c7fe7c8d737433289a65cff21df0414 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:02:11.560333 28179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c7fe7c8d737433289a65cff21df0414 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:02:11.560468 28179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c7fe7c8d737433289a65cff21df0414 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:11.561604 28179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c7fe7c8d737433289a65cff21df0414 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7fe7c8d737433289a65cff21df0414" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33691 } }
I20260929 18:02:11.561985 28179 leader_election.cc:302] T 00000000000000000000000000000000 P 3c7fe7c8d737433289a65cff21df0414 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c7fe7c8d737433289a65cff21df0414; no voters:
I20260929 18:02:11.562372 28179 leader_election.cc:288] T 00000000000000000000000000000000 P 3c7fe7c8d737433289a65cff21df0414 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:02:11.562557 28189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c7fe7c8d737433289a65cff21df0414 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:11.562847 28189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c7fe7c8d737433289a65cff21df0414 [term 1 LEADER]: Becoming Leader. State: Replica: 3c7fe7c8d737433289a65cff21df0414, State: Running, Role: LEADER
I20260929 18:02:11.563213 28189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c7fe7c8d737433289a65cff21df0414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7fe7c8d737433289a65cff21df0414" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33691 } }
I20260929 18:02:11.563747 28179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c7fe7c8d737433289a65cff21df0414 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:02:11.565073 28191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7fe7c8d737433289a65cff21df0414 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c7fe7c8d737433289a65cff21df0414. Latest consensus state: current_term: 1 leader_uuid: "3c7fe7c8d737433289a65cff21df0414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7fe7c8d737433289a65cff21df0414" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33691 } } }
I20260929 18:02:11.565218 28191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7fe7c8d737433289a65cff21df0414 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:11.565471 28190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7fe7c8d737433289a65cff21df0414 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c7fe7c8d737433289a65cff21df0414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7fe7c8d737433289a65cff21df0414" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33691 } } }
I20260929 18:02:11.565632 28190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7fe7c8d737433289a65cff21df0414 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:11.565687 28198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:02:11.569320 28198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:02:11.575837 28198 catalog_manager.cc:1347] Generated new cluster ID: d1679f8b8b234b7787f85275dd83be69
I20260929 18:02:11.575948 28198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:02:11.596074 28198 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:02:11.596464 28198 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:02:11.608584 28198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c7fe7c8d737433289a65cff21df0414: Generated new TSK 0
I20260929 18:02:11.609597 28198 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:02:11.615850 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 28111
2026-09-29T18:02:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1468 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:02:11 dist-test-slave-r7ph krb5kdc[28210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:02:11 dist-test-slave-r7ph krb5kdc[28210](info): set up 2 sockets
Sep 29 18:02:11 dist-test-slave-r7ph krb5kdc[28210](info): commencing operation
krb5kdc: starting...
W20260929 18:02:12.690649 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 18:02:12 dist-test-slave-r7ph krb5kdc[28210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:02:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:02:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:02:12.822541 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45201
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:12.938585 28223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:12.938936 28223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:12.939023 28223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:12.943264 28223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:12.943387 28223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:12.943428 28223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:12.943454 28223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:12.943497 28223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:12.949517 28223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:12.950826 28223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:12.951938 28223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:12.961978 28231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:12.961978 28229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:12.962352 28228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:12.962530 28223 server_base.cc:1034] running on GCE node
I20260929 18:02:12.963476 28223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:12.964838 28223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:12.966161 28223 hybrid_clock.cc:648] HybridClock initialized: now 1790704932966126 us; error 62 us; skew 500 ppm
Sep 29 18:02:12 dist-test-slave-r7ph krb5kdc[28210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:12.969873 28223 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:02:12.971284 28223 webserver.cc:460] Webserver started at http://127.0.0.1:42417/ using document root <none> and password file <none>
I20260929 18:02:12.971940 28223 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:12.972043 28223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:12.972289 28223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:12.974702 28223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "549ee46dbc514825a340879d2906275f"
format_stamp: "Formatted at 2026-09-29 18:02:12 on dist-test-slave-r7ph"
server_key: "9008395944857cdaa610de232d3b3e32"
server_key_iv: "5889a0e3a1d3bedcb00d3a1c8183b8b0"
server_key_version: "encryptionkey@0"
I20260929 18:02:12.975360 28223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "549ee46dbc514825a340879d2906275f"
format_stamp: "Formatted at 2026-09-29 18:02:12 on dist-test-slave-r7ph"
server_key: "9008395944857cdaa610de232d3b3e32"
server_key_iv: "5889a0e3a1d3bedcb00d3a1c8183b8b0"
server_key_version: "encryptionkey@0"
I20260929 18:02:12.979664 28223 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 18:02:12.982709 28238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:12.984323 28223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:02:12.984525 28223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "549ee46dbc514825a340879d2906275f"
format_stamp: "Formatted at 2026-09-29 18:02:12 on dist-test-slave-r7ph"
server_key: "9008395944857cdaa610de232d3b3e32"
server_key_iv: "5889a0e3a1d3bedcb00d3a1c8183b8b0"
server_key_version: "encryptionkey@0"
I20260929 18:02:12.984675 28223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:13.001245 28223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:13.005311 28223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:13.005564 28223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:13.014279 28223 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45201
I20260929 18:02:13.014468 28290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45201 every 8 connection(s)
I20260929 18:02:13.015410 28223 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:02:13.018719 28291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:13.019953 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 28223
I20260929 18:02:13.020082 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:02:13.020393 7640 external_mini_cluster.cc:1428] Setting key ba2213736eaf56f08c3af40907111418
I20260929 18:02:13.026328 28291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 549ee46dbc514825a340879d2906275f: Bootstrap starting.
I20260929 18:02:13.029351 28291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 549ee46dbc514825a340879d2906275f: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:13.030428 28291 log.cc:824] T 00000000000000000000000000000000 P 549ee46dbc514825a340879d2906275f: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:02:13 dist-test-slave-r7ph krb5kdc[28210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:02:13.032629 28291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 549ee46dbc514825a340879d2906275f: No bootstrap required, opened a new log
I20260929 18:02:13.036641 28294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:13.021963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42077 (local address 127.0.0.1:45201)
0929 18:02:13.022483 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:13.022510 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:13.022551 (+ 41us) server_negotiation.cc:407] Connection header received
0929 18:02:13.023284 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:13.023304 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:13.023710 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:13.024056 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:13.025040 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:13.026085 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:13.027304 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:13.027861 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:13.028002 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:13.031408 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:13.031439 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:13.031456 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:13.031489 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:13.033444 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:13.033991 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:13.033997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:13.034003 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:13.034134 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:13.034475 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:13.034478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:13.034480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:13.034934 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:13.035152 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:13.035483 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 18:02:13.035805 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":167,"threads_started":1}
I20260929 18:02:13.036553 28291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 549ee46dbc514825a340879d2906275f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549ee46dbc514825a340879d2906275f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45201 } }
I20260929 18:02:13.036828 28291 raft_consensus.cc:374] T 00000000000000000000000000000000 P 549ee46dbc514825a340879d2906275f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:13.036939 28291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 549ee46dbc514825a340879d2906275f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 549ee46dbc514825a340879d2906275f, State: Initialized, Role: FOLLOWER
I20260929 18:02:13.037468 28291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 549ee46dbc514825a340879d2906275f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549ee46dbc514825a340879d2906275f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45201 } }
I20260929 18:02:13.037604 28291 raft_consensus.cc:388] T 00000000000000000000000000000000 P 549ee46dbc514825a340879d2906275f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:02:13.037719 28291 raft_consensus.cc:482] T 00000000000000000000000000000000 P 549ee46dbc514825a340879d2906275f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:02:13.037832 28291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 549ee46dbc514825a340879d2906275f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:13.039081 28291 raft_consensus.cc:504] T 00000000000000000000000000000000 P 549ee46dbc514825a340879d2906275f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549ee46dbc514825a340879d2906275f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45201 } }
I20260929 18:02:13.039462 28291 leader_election.cc:302] T 00000000000000000000000000000000 P 549ee46dbc514825a340879d2906275f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 549ee46dbc514825a340879d2906275f; no voters:
I20260929 18:02:13.039819 28291 leader_election.cc:288] T 00000000000000000000000000000000 P 549ee46dbc514825a340879d2906275f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:02:13.040019 28296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 549ee46dbc514825a340879d2906275f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:13.040354 7640 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:02:13.040370 28296 raft_consensus.cc:686] T 00000000000000000000000000000000 P 549ee46dbc514825a340879d2906275f [term 1 LEADER]: Becoming Leader. State: Replica: 549ee46dbc514825a340879d2906275f, State: Running, Role: LEADER
I20260929 18:02:13.040701 28296 consensus_queue.cc:237] T 00000000000000000000000000000000 P 549ee46dbc514825a340879d2906275f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549ee46dbc514825a340879d2906275f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45201 } }
I20260929 18:02:13.041246 28291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 549ee46dbc514825a340879d2906275f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:02:13.042057 28296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 549ee46dbc514825a340879d2906275f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 549ee46dbc514825a340879d2906275f. Latest consensus state: current_term: 1 leader_uuid: "549ee46dbc514825a340879d2906275f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549ee46dbc514825a340879d2906275f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45201 } } }
I20260929 18:02:13.042176 28296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 549ee46dbc514825a340879d2906275f [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:13.042605 28309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:02:13.042565 28297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 549ee46dbc514825a340879d2906275f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "549ee46dbc514825a340879d2906275f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549ee46dbc514825a340879d2906275f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45201 } } }
I20260929 18:02:13.042728 28297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 549ee46dbc514825a340879d2906275f [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:13.046270 28309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:02:13.052875 28294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:13.042909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33439 (local address 127.0.0.1:45201)
0929 18:02:13.043018 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:13.043023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:13.043567 (+ 544us) server_negotiation.cc:407] Connection header received
0929 18:02:13.043789 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:13.043792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:13.043846 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:13.043934 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:13.044921 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:13.045702 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:13.046934 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:13.047282 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:13.047416 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:13.049185 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:13.049189 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:13.049192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:13.049223 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:13.050889 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:13.051404 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:13.051410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:13.051412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:13.051503 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:13.051819 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:13.051823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:13.051825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:13.052032 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:13.052180 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:13.052496 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 18:02:13.052654 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:02:13.054678 28309 catalog_manager.cc:1347] Generated new cluster ID: 24b8cb543639494a91de0eecbdcbc773
I20260929 18:02:13.054767 28309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:02:13.071913 28309 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:02:13.073416 28309 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:02:13.096557 28309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 549ee46dbc514825a340879d2906275f: Generated new TSK 0
I20260929 18:02:13.097558 28309 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:02:13.112234 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 28223
2026-09-29T18:02:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1496 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:02:13 dist-test-slave-r7ph krb5kdc[28322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:02:13 dist-test-slave-r7ph krb5kdc[28322](info): set up 2 sockets
Sep 29 18:02:13 dist-test-slave-r7ph krb5kdc[28322](info): commencing operation
krb5kdc: starting...
W20260929 18:02:14.190635 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 18:02:14 dist-test-slave-r7ph krb5kdc[28322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:02:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:02:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:02:14.326576 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36953
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:14.444068 28335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:14.444433 28335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:14.444522 28335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:14.448848 28335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:14.448982 28335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:14.449020 28335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:14.449061 28335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:14.449097 28335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:14.454892 28335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:14.456111 28335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:14.457239 28335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:14.466818 28340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:14.466737 28343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:14.466744 28341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:14.467095 28335 server_base.cc:1034] running on GCE node
I20260929 18:02:14.468084 28335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:14.469410 28335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:14.470582 28335 hybrid_clock.cc:648] HybridClock initialized: now 1790704934470576 us; error 44 us; skew 500 ppm
Sep 29 18:02:14 dist-test-slave-r7ph krb5kdc[28322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:14.473791 28335 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:02:14.474951 28335 webserver.cc:460] Webserver started at http://127.0.0.1:33791/ using document root <none> and password file <none>
I20260929 18:02:14.475548 28335 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:14.475615 28335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:14.475884 28335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:14.478130 28335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "4dfe1864439b467e8b5a7cfab5284db6"
format_stamp: "Formatted at 2026-09-29 18:02:14 on dist-test-slave-r7ph"
server_key: "b233707a6c125e6a6c52da2e0fe3300d"
server_key_iv: "0c518686f90121a210f3df54aeafffcc"
server_key_version: "encryptionkey@0"
I20260929 18:02:14.478809 28335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "4dfe1864439b467e8b5a7cfab5284db6"
format_stamp: "Formatted at 2026-09-29 18:02:14 on dist-test-slave-r7ph"
server_key: "b233707a6c125e6a6c52da2e0fe3300d"
server_key_iv: "0c518686f90121a210f3df54aeafffcc"
server_key_version: "encryptionkey@0"
I20260929 18:02:14.482913 28335 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:02:14.485773 28350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:14.487287 28335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:02:14.487491 28335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "4dfe1864439b467e8b5a7cfab5284db6"
format_stamp: "Formatted at 2026-09-29 18:02:14 on dist-test-slave-r7ph"
server_key: "b233707a6c125e6a6c52da2e0fe3300d"
server_key_iv: "0c518686f90121a210f3df54aeafffcc"
server_key_version: "encryptionkey@0"
I20260929 18:02:14.487639 28335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:14.520987 28335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:14.524855 28335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:14.525098 28335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:14.533291 28335 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36953
I20260929 18:02:14.533337 28402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36953 every 8 connection(s)
I20260929 18:02:14.534420 28335 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:02:14.537475 28403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:14.543618 28403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dfe1864439b467e8b5a7cfab5284db6: Bootstrap starting.
I20260929 18:02:14.544559 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 28335
I20260929 18:02:14.544718 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:02:14.545022 7640 external_mini_cluster.cc:1428] Setting key 98195a50463874404678f00425c91a27
I20260929 18:02:14.548098 28403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dfe1864439b467e8b5a7cfab5284db6: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:14.548935 28403 log.cc:824] T 00000000000000000000000000000000 P 4dfe1864439b467e8b5a7cfab5284db6: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:14.551514 28403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dfe1864439b467e8b5a7cfab5284db6: No bootstrap required, opened a new log
I20260929 18:02:14.556213 28406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:14.546812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48289 (local address 127.0.0.1:36953)
0929 18:02:14.547428 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:14.547450 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:14.547500 (+ 50us) server_negotiation.cc:407] Connection header received
0929 18:02:14.548453 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:14.548477 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:14.548880 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:14.549344 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:14.549872 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:14.551002 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:14.552268 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:14.552848 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:14.553003 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:14.553977 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:14.553983 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:14.553999 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:14.554589 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:14.554657 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:14.554685 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:14.554820 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:14.555020 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 18:02:14.555307 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":148,"threads_started":1}
I20260929 18:02:14.556043 28403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dfe1864439b467e8b5a7cfab5284db6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfe1864439b467e8b5a7cfab5284db6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36953 } }
I20260929 18:02:14.556459 28403 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dfe1864439b467e8b5a7cfab5284db6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:14.556577 28403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dfe1864439b467e8b5a7cfab5284db6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dfe1864439b467e8b5a7cfab5284db6, State: Initialized, Role: FOLLOWER
I20260929 18:02:14.557137 28403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dfe1864439b467e8b5a7cfab5284db6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfe1864439b467e8b5a7cfab5284db6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36953 } }
I20260929 18:02:14.557310 28403 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dfe1864439b467e8b5a7cfab5284db6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:02:14.557444 28403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dfe1864439b467e8b5a7cfab5284db6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:02:14.557571 28403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dfe1864439b467e8b5a7cfab5284db6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:14.558866 28403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dfe1864439b467e8b5a7cfab5284db6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfe1864439b467e8b5a7cfab5284db6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36953 } }
I20260929 18:02:14.559291 28403 leader_election.cc:302] T 00000000000000000000000000000000 P 4dfe1864439b467e8b5a7cfab5284db6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dfe1864439b467e8b5a7cfab5284db6; no voters:
I20260929 18:02:14.559674 28403 leader_election.cc:288] T 00000000000000000000000000000000 P 4dfe1864439b467e8b5a7cfab5284db6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:02:14.559746 7640 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:02:14.559787 28408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dfe1864439b467e8b5a7cfab5284db6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:14.560040 28408 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dfe1864439b467e8b5a7cfab5284db6 [term 1 LEADER]: Becoming Leader. State: Replica: 4dfe1864439b467e8b5a7cfab5284db6, State: Running, Role: LEADER
I20260929 18:02:14.560446 28408 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dfe1864439b467e8b5a7cfab5284db6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfe1864439b467e8b5a7cfab5284db6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36953 } }
I20260929 18:02:14.561015 28403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dfe1864439b467e8b5a7cfab5284db6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:02:14.561795 28408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dfe1864439b467e8b5a7cfab5284db6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dfe1864439b467e8b5a7cfab5284db6. Latest consensus state: current_term: 1 leader_uuid: "4dfe1864439b467e8b5a7cfab5284db6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfe1864439b467e8b5a7cfab5284db6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36953 } } }
I20260929 18:02:14.561954 28408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dfe1864439b467e8b5a7cfab5284db6 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:14.562345 28409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dfe1864439b467e8b5a7cfab5284db6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dfe1864439b467e8b5a7cfab5284db6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfe1864439b467e8b5a7cfab5284db6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36953 } } }
I20260929 18:02:14.562474 28409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dfe1864439b467e8b5a7cfab5284db6 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:14.562587 28420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:02:14.566083 28420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:02:14.573328 28420 catalog_manager.cc:1347] Generated new cluster ID: 98c954ed7cac427990e3ded36870a8c7
I20260929 18:02:14.573422 28420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:02:14.573920 28406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:14.568802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33585 (local address 127.0.0.1:36953)
0929 18:02:14.568927 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:14.568931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:14.569262 (+ 331us) server_negotiation.cc:407] Connection header received
0929 18:02:14.569468 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:14.569472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:14.569525 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:14.569611 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:14.569991 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:14.570736 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:14.571777 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:14.572026 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:14.572159 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:14.573365 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:14.573369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:14.573372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:14.573400 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:14.573456 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:14.573476 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:14.573549 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:14.573698 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 18:02:14.573760 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:02:14.599177 28420 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:02:14.600224 28420 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:02:14.608929 28420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dfe1864439b467e8b5a7cfab5284db6: Generated new TSK 0
I20260929 18:02:14.609879 28420 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:02:14.620060 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 28335
2026-09-29T18:02:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1507 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:02:14 dist-test-slave-r7ph krb5kdc[28434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:02:14 dist-test-slave-r7ph krb5kdc[28434](info): set up 2 sockets
Sep 29 18:02:14 dist-test-slave-r7ph krb5kdc[28434](info): commencing operation
krb5kdc: starting...
W20260929 18:02:15.690682 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 18:02:15 dist-test-slave-r7ph krb5kdc[28434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:02:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:02:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:02:15.831030 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43145
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:15.950670 28447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:15.951036 28447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:15.951083 28447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:15.955472 28447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:15.955554 28447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:15.955574 28447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:15.955592 28447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:15.955607 28447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:15.961498 28447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:15.962829 28447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:15.963963 28447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:15.973992 28453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:15.974225 28452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:15.974419 28455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:15.974500 28447 server_base.cc:1034] running on GCE node
I20260929 18:02:15.975163 28447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:15.976498 28447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:15.977737 28447 hybrid_clock.cc:648] HybridClock initialized: now 1790704935977710 us; error 39 us; skew 500 ppm
Sep 29 18:02:15 dist-test-slave-r7ph krb5kdc[28434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:15.981338 28447 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:02:15.982627 28447 webserver.cc:460] Webserver started at http://127.0.0.1:33001/ using document root <none> and password file <none>
I20260929 18:02:15.983243 28447 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:15.983312 28447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:15.983496 28447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:15.985726 28447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "c4d632c85773422693ea7722436ce61d"
format_stamp: "Formatted at 2026-09-29 18:02:15 on dist-test-slave-r7ph"
server_key: "9d23f110bb2c28c9ebfc4e534be42d7e"
server_key_iv: "a0b5112370232afb38afce03b181b464"
server_key_version: "encryptionkey@0"
I20260929 18:02:15.986446 28447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "c4d632c85773422693ea7722436ce61d"
format_stamp: "Formatted at 2026-09-29 18:02:15 on dist-test-slave-r7ph"
server_key: "9d23f110bb2c28c9ebfc4e534be42d7e"
server_key_iv: "a0b5112370232afb38afce03b181b464"
server_key_version: "encryptionkey@0"
I20260929 18:02:15.990833 28447 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:02:15.993841 28462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:15.995556 28447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:02:15.995728 28447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "c4d632c85773422693ea7722436ce61d"
format_stamp: "Formatted at 2026-09-29 18:02:15 on dist-test-slave-r7ph"
server_key: "9d23f110bb2c28c9ebfc4e534be42d7e"
server_key_iv: "a0b5112370232afb38afce03b181b464"
server_key_version: "encryptionkey@0"
I20260929 18:02:15.995862 28447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:16.001488 28447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:16.005316 28447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:16.005497 28447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:16.013623 28447 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43145
I20260929 18:02:16.013646 28514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43145 every 8 connection(s)
I20260929 18:02:16.014722 28447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:02:16.017973 28515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:16.018538 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 28447
I20260929 18:02:16.018690 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:02:16.019001 7640 external_mini_cluster.cc:1428] Setting key b709db3a910602e3c1d6647961ce0754
I20260929 18:02:16.025353 28518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:16.020727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42771 (local address 127.0.0.1:43145)
0929 18:02:16.021242 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:16.021260 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:16.021299 (+ 39us) server_negotiation.cc:407] Connection header received
0929 18:02:16.022077 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:16.022102 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:16.022548 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:16.022876 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:16.023869 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:16.023874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:16.023887 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:16.023898 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:16.023956 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:16.023976 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:16.024099 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:16.024233 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 18:02:16.024576 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":147,"threads_started":1}
I20260929 18:02:16.026099 28515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d632c85773422693ea7722436ce61d: Bootstrap starting.
I20260929 18:02:16.028578 7640 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:02:16.028584 28515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4d632c85773422693ea7722436ce61d: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:16.029541 28515 log.cc:824] T 00000000000000000000000000000000 P c4d632c85773422693ea7722436ce61d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:16.032334 28515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d632c85773422693ea7722436ce61d: No bootstrap required, opened a new log
I20260929 18:02:16.033093 28518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:16.031151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37507 (local address 127.0.0.1:43145)
0929 18:02:16.031335 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:16.031340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:16.031615 (+ 275us) server_negotiation.cc:407] Connection header received
0929 18:02:16.031730 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:16.031733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:16.031768 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:16.031836 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:16.032644 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:16.032647 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:16.032650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:16.032660 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:16.032688 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:16.032699 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:16.032779 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:16.032898 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 18:02:16.032934 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:02:16.036379 28515 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4d632c85773422693ea7722436ce61d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d632c85773422693ea7722436ce61d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43145 } }
I20260929 18:02:16.036675 28515 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4d632c85773422693ea7722436ce61d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:16.036743 28515 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4d632c85773422693ea7722436ce61d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d632c85773422693ea7722436ce61d, State: Initialized, Role: FOLLOWER
I20260929 18:02:16.037245 28515 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4d632c85773422693ea7722436ce61d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d632c85773422693ea7722436ce61d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43145 } }
I20260929 18:02:16.037402 28515 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4d632c85773422693ea7722436ce61d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:02:16.037501 28515 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4d632c85773422693ea7722436ce61d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:02:16.037608 28515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4d632c85773422693ea7722436ce61d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:16.038720 28515 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4d632c85773422693ea7722436ce61d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d632c85773422693ea7722436ce61d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43145 } }
I20260929 18:02:16.039100 28515 leader_election.cc:302] T 00000000000000000000000000000000 P c4d632c85773422693ea7722436ce61d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4d632c85773422693ea7722436ce61d; no voters:
I20260929 18:02:16.039451 28515 leader_election.cc:288] T 00000000000000000000000000000000 P c4d632c85773422693ea7722436ce61d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:02:16.039738 28525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4d632c85773422693ea7722436ce61d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:16.040004 28525 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4d632c85773422693ea7722436ce61d [term 1 LEADER]: Becoming Leader. State: Replica: c4d632c85773422693ea7722436ce61d, State: Running, Role: LEADER
I20260929 18:02:16.040385 28525 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4d632c85773422693ea7722436ce61d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d632c85773422693ea7722436ce61d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43145 } }
I20260929 18:02:16.041009 28515 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4d632c85773422693ea7722436ce61d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:02:16.042564 28527 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d632c85773422693ea7722436ce61d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4d632c85773422693ea7722436ce61d. Latest consensus state: current_term: 1 leader_uuid: "c4d632c85773422693ea7722436ce61d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d632c85773422693ea7722436ce61d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43145 } } }
I20260929 18:02:16.042702 28527 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d632c85773422693ea7722436ce61d [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:16.042670 28526 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d632c85773422693ea7722436ce61d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4d632c85773422693ea7722436ce61d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d632c85773422693ea7722436ce61d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43145 } } }
I20260929 18:02:16.042786 28526 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d632c85773422693ea7722436ce61d [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:16.043219 28534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:02:16.046070 28534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:02:16.052016 28534 catalog_manager.cc:1347] Generated new cluster ID: b14da3f598704f82bc2cbfebad092db3
I20260929 18:02:16.052120 28534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:02:16.065833 28534 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:02:16.066320 28534 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:02:16.081395 28534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4d632c85773422693ea7722436ce61d: Generated new TSK 0
I20260929 18:02:16.082425 28534 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:02:16.088708 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 28447
2026-09-29T18:02:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1468 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:02:16 dist-test-slave-r7ph krb5kdc[28546](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:02:16 dist-test-slave-r7ph krb5kdc[28546](info): set up 2 sockets
Sep 29 18:02:16 dist-test-slave-r7ph krb5kdc[28546](info): commencing operation
krb5kdc: starting...
W20260929 18:02:17.195366 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 18:02:17 dist-test-slave-r7ph krb5kdc[28546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:02:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:02:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:02:17.334508 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35843
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:17.460265 28559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:17.460685 28559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:17.460777 28559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:17.465605 28559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:17.465713 28559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:17.465736 28559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:17.465756 28559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:17.465773 28559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:17.472391 28559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:17.473560 28559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:17.474687 28559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:17.484740 28565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:17.484776 28564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:17.484747 28567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:17.484813 28559 server_base.cc:1034] running on GCE node
I20260929 18:02:17.485667 28559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:17.487042 28559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:17.488241 28559 hybrid_clock.cc:648] HybridClock initialized: now 1790704937488225 us; error 44 us; skew 500 ppm
Sep 29 18:02:17 dist-test-slave-r7ph krb5kdc[28546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:17.502929 28559 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:02:17.504153 28559 webserver.cc:460] Webserver started at http://127.0.0.1:32823/ using document root <none> and password file <none>
I20260929 18:02:17.504808 28559 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:17.504879 28559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:17.505134 28559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:17.507918 28559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "efa2a7f7484f430aaa48f2859d8e2af7"
format_stamp: "Formatted at 2026-09-29 18:02:17 on dist-test-slave-r7ph"
server_key: "05a33c7b19dd138b3e5d1f2d1ebca142"
server_key_iv: "3f364a3b5b5564eb4e3ef3eeb1398876"
server_key_version: "encryptionkey@0"
I20260929 18:02:17.508610 28559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "efa2a7f7484f430aaa48f2859d8e2af7"
format_stamp: "Formatted at 2026-09-29 18:02:17 on dist-test-slave-r7ph"
server_key: "05a33c7b19dd138b3e5d1f2d1ebca142"
server_key_iv: "3f364a3b5b5564eb4e3ef3eeb1398876"
server_key_version: "encryptionkey@0"
I20260929 18:02:17.513336 28559 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:02:17.516369 28574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:17.517804 28559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:02:17.518077 28559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "efa2a7f7484f430aaa48f2859d8e2af7"
format_stamp: "Formatted at 2026-09-29 18:02:17 on dist-test-slave-r7ph"
server_key: "05a33c7b19dd138b3e5d1f2d1ebca142"
server_key_iv: "3f364a3b5b5564eb4e3ef3eeb1398876"
server_key_version: "encryptionkey@0"
I20260929 18:02:17.518218 28559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:17.532948 28559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:17.537120 28559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:17.537389 28559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:17.545264 28559 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35843
I20260929 18:02:17.545280 28626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35843 every 8 connection(s)
I20260929 18:02:17.546418 28559 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:02:17.549490 28627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:17.552470 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 28559
I20260929 18:02:17.552635 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:02:17.552981 7640 external_mini_cluster.cc:1428] Setting key 2f89165133f739a11477350734968b68
I20260929 18:02:17.556563 28627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efa2a7f7484f430aaa48f2859d8e2af7: Bootstrap starting.
I20260929 18:02:17.559999 28627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efa2a7f7484f430aaa48f2859d8e2af7: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:17.561082 28627 log.cc:824] T 00000000000000000000000000000000 P efa2a7f7484f430aaa48f2859d8e2af7: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:17.563843 28627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efa2a7f7484f430aaa48f2859d8e2af7: No bootstrap required, opened a new log
Sep 29 18:02:17 dist-test-slave-r7ph krb5kdc[28546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:02:17.567648 28627 raft_consensus.cc:348] T 00000000000000000000000000000000 P efa2a7f7484f430aaa48f2859d8e2af7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa2a7f7484f430aaa48f2859d8e2af7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35843 } }
I20260929 18:02:17.567955 28627 raft_consensus.cc:374] T 00000000000000000000000000000000 P efa2a7f7484f430aaa48f2859d8e2af7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:17.568053 28627 raft_consensus.cc:729] T 00000000000000000000000000000000 P efa2a7f7484f430aaa48f2859d8e2af7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa2a7f7484f430aaa48f2859d8e2af7, State: Initialized, Role: FOLLOWER
I20260929 18:02:17.568595 28627 consensus_queue.cc:260] T 00000000000000000000000000000000 P efa2a7f7484f430aaa48f2859d8e2af7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa2a7f7484f430aaa48f2859d8e2af7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35843 } }
I20260929 18:02:17.568769 28627 raft_consensus.cc:388] T 00000000000000000000000000000000 P efa2a7f7484f430aaa48f2859d8e2af7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:02:17.568881 28627 raft_consensus.cc:482] T 00000000000000000000000000000000 P efa2a7f7484f430aaa48f2859d8e2af7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:02:17.568988 28627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efa2a7f7484f430aaa48f2859d8e2af7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:17.570371 28627 raft_consensus.cc:504] T 00000000000000000000000000000000 P efa2a7f7484f430aaa48f2859d8e2af7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa2a7f7484f430aaa48f2859d8e2af7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35843 } }
I20260929 18:02:17.570565 28630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:17.554621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:58281 (local address 127.0.0.1:35843)
0929 18:02:17.555082 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:17.555105 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:17.555152 (+ 47us) server_negotiation.cc:407] Connection header received
0929 18:02:17.555823 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:17.555842 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:17.556227 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:17.556541 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:17.557492 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:17.559274 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:17.560663 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:17.561447 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:17.561618 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:17.564609 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:17.564616 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:17.564632 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:17.564665 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:17.566718 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:17.567362 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:17.567368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:17.567374 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:17.567506 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:17.567879 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:17.567889 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:17.567891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:17.568427 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:17.568688 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:17.569057 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 18:02:17.569345 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":148,"threads_started":1}
I20260929 18:02:17.570797 28627 leader_election.cc:302] T 00000000000000000000000000000000 P efa2a7f7484f430aaa48f2859d8e2af7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efa2a7f7484f430aaa48f2859d8e2af7; no voters:
I20260929 18:02:17.571151 28627 leader_election.cc:288] T 00000000000000000000000000000000 P efa2a7f7484f430aaa48f2859d8e2af7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:02:17.571297 28632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efa2a7f7484f430aaa48f2859d8e2af7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:17.571534 28632 raft_consensus.cc:686] T 00000000000000000000000000000000 P efa2a7f7484f430aaa48f2859d8e2af7 [term 1 LEADER]: Becoming Leader. State: Replica: efa2a7f7484f430aaa48f2859d8e2af7, State: Running, Role: LEADER
I20260929 18:02:17.571972 28632 consensus_queue.cc:237] T 00000000000000000000000000000000 P efa2a7f7484f430aaa48f2859d8e2af7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa2a7f7484f430aaa48f2859d8e2af7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35843 } }
I20260929 18:02:17.572537 28627 sys_catalog.cc:564] T 00000000000000000000000000000000 P efa2a7f7484f430aaa48f2859d8e2af7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:02:17.573354 28632 sys_catalog.cc:455] T 00000000000000000000000000000000 P efa2a7f7484f430aaa48f2859d8e2af7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efa2a7f7484f430aaa48f2859d8e2af7. Latest consensus state: current_term: 1 leader_uuid: "efa2a7f7484f430aaa48f2859d8e2af7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa2a7f7484f430aaa48f2859d8e2af7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35843 } } }
I20260929 18:02:17.573508 28632 sys_catalog.cc:458] T 00000000000000000000000000000000 P efa2a7f7484f430aaa48f2859d8e2af7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:17.574143 28633 sys_catalog.cc:455] T 00000000000000000000000000000000 P efa2a7f7484f430aaa48f2859d8e2af7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efa2a7f7484f430aaa48f2859d8e2af7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa2a7f7484f430aaa48f2859d8e2af7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35843 } } }
I20260929 18:02:17.574362 28633 sys_catalog.cc:458] T 00000000000000000000000000000000 P efa2a7f7484f430aaa48f2859d8e2af7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:17.574472 7640 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260929 18:02:17.578346 28647 catalog_manager.cc:1558] T 00000000000000000000000000000000 P efa2a7f7484f430aaa48f2859d8e2af7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:02:17.578440 28647 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:02:17.578508 28641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:02:17.579279 28641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:02:17.586817 28641 catalog_manager.cc:1347] Generated new cluster ID: 9b4a5feb58c048a28eec3ad39be7387d
I20260929 18:02:17.586900 28641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:02:17.589133 28630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:17.577099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:57079 (local address 127.0.0.1:35843)
0929 18:02:17.577225 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:17.577232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:17.577462 (+ 230us) server_negotiation.cc:407] Connection header received
0929 18:02:17.577612 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:17.577615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:17.577672 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:17.577761 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:17.578677 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:17.579580 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:17.580747 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:17.581174 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:17.581294 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:17.583301 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:17.583308 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:17.583313 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:17.583354 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:17.585997 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:17.587157 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:17.587161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:17.587163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:17.587265 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:17.587681 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:17.587685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:17.587687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:17.587918 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:17.588123 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:17.588842 (+ 719us) server_negotiation.cc:299] Negotiation successful
0929 18:02:17.588950 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:02:17.597993 28641 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:02:17.599172 28641 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:02:17.604930 28641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efa2a7f7484f430aaa48f2859d8e2af7: Generated new TSK 0
I20260929 18:02:17.605787 28641 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:02:17.614136 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 28559
W20260929 18:02:17.619616 28544 connection.cc:535] client connection to 127.0.0.1:35843 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35843): Connection reset by peer (error 104)
2026-09-29T18:02:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1525 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:02:17 dist-test-slave-r7ph krb5kdc[28658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:02:17 dist-test-slave-r7ph krb5kdc[28658](info): set up 2 sockets
Sep 29 18:02:17 dist-test-slave-r7ph krb5kdc[28658](info): commencing operation
krb5kdc: starting...
W20260929 18:02:18.694625 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 18:02:18 dist-test-slave-r7ph krb5kdc[28658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:02:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:02:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:02:18.833873 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38871
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:18.949198 28671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:18.949587 28671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:18.949662 28671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:18.953919 28671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:18.954007 28671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:18.954027 28671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:18.954046 28671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:18.954061 28671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:18.959831 28671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:18.960951 28671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:18.961957 28671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:18.971711 28679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:18.971664 28676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:18.971665 28677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:18.972196 28671 server_base.cc:1034] running on GCE node
I20260929 18:02:18.973018 28671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:18.974529 28671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:18.975754 28671 hybrid_clock.cc:648] HybridClock initialized: now 1790704938975740 us; error 36 us; skew 500 ppm
Sep 29 18:02:18 dist-test-slave-r7ph krb5kdc[28658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:18.979326 28671 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:02:18.980587 28671 webserver.cc:460] Webserver started at http://127.0.0.1:44017/ using document root <none> and password file <none>
I20260929 18:02:18.981226 28671 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:18.981304 28671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:18.981499 28671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:18.984141 28671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "77be339c1e05400189f740bb379e0191"
format_stamp: "Formatted at 2026-09-29 18:02:18 on dist-test-slave-r7ph"
server_key: "55d49d69e5dbb201ff5775a3ae25f206"
server_key_iv: "c73add7ce452cdb4af44b113a345d40f"
server_key_version: "encryptionkey@0"
I20260929 18:02:18.984835 28671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "77be339c1e05400189f740bb379e0191"
format_stamp: "Formatted at 2026-09-29 18:02:18 on dist-test-slave-r7ph"
server_key: "55d49d69e5dbb201ff5775a3ae25f206"
server_key_iv: "c73add7ce452cdb4af44b113a345d40f"
server_key_version: "encryptionkey@0"
I20260929 18:02:18.989662 28671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 18:02:18.992762 28686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:18.994683 28671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 18:02:18.994843 28671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "77be339c1e05400189f740bb379e0191"
format_stamp: "Formatted at 2026-09-29 18:02:18 on dist-test-slave-r7ph"
server_key: "55d49d69e5dbb201ff5775a3ae25f206"
server_key_iv: "c73add7ce452cdb4af44b113a345d40f"
server_key_version: "encryptionkey@0"
I20260929 18:02:18.994978 28671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:19.011299 28671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:19.014925 28671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:19.015105 28671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:19.023061 28671 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38871
I20260929 18:02:19.023097 28738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38871 every 8 connection(s)
I20260929 18:02:19.024150 28671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:02:19.027449 28739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:19.031270 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 28671
I20260929 18:02:19.031468 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:02:19.031811 7640 external_mini_cluster.cc:1428] Setting key 7ffeb743cff1982bd57d5f89840fd82c
I20260929 18:02:19.034358 28739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77be339c1e05400189f740bb379e0191: Bootstrap starting.
I20260929 18:02:19.036986 28739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77be339c1e05400189f740bb379e0191: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:19.037809 28739 log.cc:824] T 00000000000000000000000000000000 P 77be339c1e05400189f740bb379e0191: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:19.040164 28739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77be339c1e05400189f740bb379e0191: No bootstrap required, opened a new log
I20260929 18:02:19.041863 28742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:19.033514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:35415 (local address 127.0.0.1:38871)
0929 18:02:19.033985 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:19.034005 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:19.034044 (+ 39us) server_negotiation.cc:407] Connection header received
0929 18:02:19.034840 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:19.034862 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:19.035271 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:19.035576 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:19.035981 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:19.037014 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:19.037995 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:19.038642 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:19.038836 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:19.039883 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:19.039888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:19.039903 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:19.040453 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:19.040519 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:19.040545 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:19.040701 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:19.040904 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 18:02:19.041064 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":150,"threads_started":1}
I20260929 18:02:19.044929 28739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77be339c1e05400189f740bb379e0191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77be339c1e05400189f740bb379e0191" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38871 } }
I20260929 18:02:19.045284 28739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77be339c1e05400189f740bb379e0191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:19.045341 28739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77be339c1e05400189f740bb379e0191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77be339c1e05400189f740bb379e0191, State: Initialized, Role: FOLLOWER
I20260929 18:02:19.045761 7640 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:02:19.045878 28739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77be339c1e05400189f740bb379e0191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77be339c1e05400189f740bb379e0191" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38871 } }
I20260929 18:02:19.046058 28739 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77be339c1e05400189f740bb379e0191 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:02:19.046132 28739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77be339c1e05400189f740bb379e0191 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:02:19.046241 28739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77be339c1e05400189f740bb379e0191 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:19.047461 28739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77be339c1e05400189f740bb379e0191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77be339c1e05400189f740bb379e0191" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38871 } }
I20260929 18:02:19.048091 28739 leader_election.cc:302] T 00000000000000000000000000000000 P 77be339c1e05400189f740bb379e0191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77be339c1e05400189f740bb379e0191; no voters:
I20260929 18:02:19.048540 28739 leader_election.cc:288] T 00000000000000000000000000000000 P 77be339c1e05400189f740bb379e0191 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:02:19.048655 28749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77be339c1e05400189f740bb379e0191 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:19.048949 28749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77be339c1e05400189f740bb379e0191 [term 1 LEADER]: Becoming Leader. State: Replica: 77be339c1e05400189f740bb379e0191, State: Running, Role: LEADER
I20260929 18:02:19.049417 28749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77be339c1e05400189f740bb379e0191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77be339c1e05400189f740bb379e0191" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38871 } }
I20260929 18:02:19.050292 28739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77be339c1e05400189f740bb379e0191 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:02:19.051317 28750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77be339c1e05400189f740bb379e0191 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77be339c1e05400189f740bb379e0191" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77be339c1e05400189f740bb379e0191" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38871 } } }
I20260929 18:02:19.051527 28750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77be339c1e05400189f740bb379e0191 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:19.051904 28751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77be339c1e05400189f740bb379e0191 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77be339c1e05400189f740bb379e0191. Latest consensus state: current_term: 1 leader_uuid: "77be339c1e05400189f740bb379e0191" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77be339c1e05400189f740bb379e0191" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38871 } } }
I20260929 18:02:19.052060 28751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77be339c1e05400189f740bb379e0191 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:19.053438 28758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:02:19.053896 28742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:19.048217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:37909 (local address 127.0.0.1:38871)
0929 18:02:19.048365 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:19.048371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:19.048629 (+ 258us) server_negotiation.cc:407] Connection header received
0929 18:02:19.048814 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:19.048818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:19.048912 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:19.049011 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:19.049437 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:19.050279 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:19.051542 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:19.051940 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:19.052028 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:19.053409 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:19.053413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:19.053418 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:19.053435 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:19.053464 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:19.053478 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:19.053593 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:19.053720 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 18:02:19.053756 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:02:19.056654 28758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:02:19.063280 28758 catalog_manager.cc:1347] Generated new cluster ID: c6fbb11d0cdf49e4a26737e2ed3a9841
I20260929 18:02:19.063386 28758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:02:19.079561 28758 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:02:19.080966 28758 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:02:19.088606 28758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77be339c1e05400189f740bb379e0191: Generated new TSK 0
I20260929 18:02:19.089605 28758 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:02:19.096877 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 28671
W20260929 18:02:19.102429 28656 connection.cc:535] client connection to 127.0.0.1:38871 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38871): Connection reset by peer (error 104)
2026-09-29T18:02:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1482 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:02:19 dist-test-slave-r7ph krb5kdc[28770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:02:19 dist-test-slave-r7ph krb5kdc[28770](info): set up 2 sockets
Sep 29 18:02:19 dist-test-slave-r7ph krb5kdc[28770](info): commencing operation
krb5kdc: starting...
W20260929 18:02:20.182731 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 18:02:20 dist-test-slave-r7ph krb5kdc[28770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:02:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:02:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:02:20.321919 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45905
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:20.441438 28783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:20.441751 28783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:20.441795 28783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:20.446063 28783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:20.446151 28783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:20.446171 28783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:20.446188 28783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:20.446204 28783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:20.451922 28783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:20.453138 28783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:20.454224 28783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:20.463462 28791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:20.463460 28789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:20.463500 28788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:20.463712 28783 server_base.cc:1034] running on GCE node
I20260929 18:02:20.464463 28783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:20.465718 28783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:20.466883 28783 hybrid_clock.cc:648] HybridClock initialized: now 1790704940466873 us; error 32 us; skew 500 ppm
Sep 29 18:02:20 dist-test-slave-r7ph krb5kdc[28770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:20.470122 28783 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:02:20.471274 28783 webserver.cc:460] Webserver started at http://127.0.0.1:33453/ using document root <none> and password file <none>
I20260929 18:02:20.471892 28783 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:20.471989 28783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:20.472257 28783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:20.474634 28783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "359c5d1eac914266be49ff47cd7a3a2c"
format_stamp: "Formatted at 2026-09-29 18:02:20 on dist-test-slave-r7ph"
server_key: "f08e87f308df1504c00c6c88702d0aae"
server_key_iv: "f92eb00da1e585df0dc5623b9d2061e1"
server_key_version: "encryptionkey@0"
I20260929 18:02:20.475293 28783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "359c5d1eac914266be49ff47cd7a3a2c"
format_stamp: "Formatted at 2026-09-29 18:02:20 on dist-test-slave-r7ph"
server_key: "f08e87f308df1504c00c6c88702d0aae"
server_key_iv: "f92eb00da1e585df0dc5623b9d2061e1"
server_key_version: "encryptionkey@0"
I20260929 18:02:20.479429 28783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 18:02:20.482132 28798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:20.483482 28783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:02:20.483662 28783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "359c5d1eac914266be49ff47cd7a3a2c"
format_stamp: "Formatted at 2026-09-29 18:02:20 on dist-test-slave-r7ph"
server_key: "f08e87f308df1504c00c6c88702d0aae"
server_key_iv: "f92eb00da1e585df0dc5623b9d2061e1"
server_key_version: "encryptionkey@0"
I20260929 18:02:20.483800 28783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:20.489012 28783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:20.492709 28783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:20.492936 28783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:20.500602 28783 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45905
I20260929 18:02:20.500639 28850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45905 every 8 connection(s)
I20260929 18:02:20.501657 28783 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:02:20.504695 28851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:20.509029 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 28783
I20260929 18:02:20.509191 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:02:20.509506 7640 external_mini_cluster.cc:1428] Setting key daa4add922f53f2eea2646a25a072084
I20260929 18:02:20.511261 28851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 359c5d1eac914266be49ff47cd7a3a2c: Bootstrap starting.
I20260929 18:02:20.514005 28851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 359c5d1eac914266be49ff47cd7a3a2c: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:20.514906 28851 log.cc:824] T 00000000000000000000000000000000 P 359c5d1eac914266be49ff47cd7a3a2c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:20.515282 28854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:20.510976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:47425 (local address 127.0.0.1:45905)
0929 18:02:20.511478 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:20.511493 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:20.511533 (+ 40us) server_negotiation.cc:407] Connection header received
0929 18:02:20.512209 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:20.512227 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:20.512528 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:20.512902 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:20.513786 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:20.513791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:20.513803 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:20.513814 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:20.513869 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:20.513888 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:20.514003 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:20.514133 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 18:02:20.514474 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":181,"threads_started":1}
I20260929 18:02:20.517204 28851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 359c5d1eac914266be49ff47cd7a3a2c: No bootstrap required, opened a new log
I20260929 18:02:20.518391 7640 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:02:20.521906 28851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 359c5d1eac914266be49ff47cd7a3a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "359c5d1eac914266be49ff47cd7a3a2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45905 } }
I20260929 18:02:20.522277 28854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:20.520541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:47991 (local address 127.0.0.1:45905)
0929 18:02:20.520676 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:20.520686 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:20.520846 (+ 160us) server_negotiation.cc:407] Connection header received
0929 18:02:20.520986 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:20.520988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:20.521021 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:20.521081 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:20.521829 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:20.521832 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:20.521834 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:20.521845 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:20.521872 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:20.521882 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:20.521957 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:20.522072 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 18:02:20.522110 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:02:20.522367 28851 raft_consensus.cc:374] T 00000000000000000000000000000000 P 359c5d1eac914266be49ff47cd7a3a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:20.522442 28851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 359c5d1eac914266be49ff47cd7a3a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359c5d1eac914266be49ff47cd7a3a2c, State: Initialized, Role: FOLLOWER
I20260929 18:02:20.522922 28851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 359c5d1eac914266be49ff47cd7a3a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "359c5d1eac914266be49ff47cd7a3a2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45905 } }
I20260929 18:02:20.523061 28851 raft_consensus.cc:388] T 00000000000000000000000000000000 P 359c5d1eac914266be49ff47cd7a3a2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:02:20.523152 28851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 359c5d1eac914266be49ff47cd7a3a2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:02:20.523260 28851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 359c5d1eac914266be49ff47cd7a3a2c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:20.524292 28851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 359c5d1eac914266be49ff47cd7a3a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "359c5d1eac914266be49ff47cd7a3a2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45905 } }
I20260929 18:02:20.524673 28851 leader_election.cc:302] T 00000000000000000000000000000000 P 359c5d1eac914266be49ff47cd7a3a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 359c5d1eac914266be49ff47cd7a3a2c; no voters:
I20260929 18:02:20.524999 28851 leader_election.cc:288] T 00000000000000000000000000000000 P 359c5d1eac914266be49ff47cd7a3a2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:02:20.525157 28861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 359c5d1eac914266be49ff47cd7a3a2c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:20.525395 28861 raft_consensus.cc:686] T 00000000000000000000000000000000 P 359c5d1eac914266be49ff47cd7a3a2c [term 1 LEADER]: Becoming Leader. State: Replica: 359c5d1eac914266be49ff47cd7a3a2c, State: Running, Role: LEADER
I20260929 18:02:20.525748 28861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 359c5d1eac914266be49ff47cd7a3a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "359c5d1eac914266be49ff47cd7a3a2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45905 } }
I20260929 18:02:20.526368 28851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 359c5d1eac914266be49ff47cd7a3a2c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:02:20.527431 28863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 359c5d1eac914266be49ff47cd7a3a2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 359c5d1eac914266be49ff47cd7a3a2c. Latest consensus state: current_term: 1 leader_uuid: "359c5d1eac914266be49ff47cd7a3a2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "359c5d1eac914266be49ff47cd7a3a2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45905 } } }
I20260929 18:02:20.527565 28863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 359c5d1eac914266be49ff47cd7a3a2c [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:20.527964 28870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:02:20.528834 28862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 359c5d1eac914266be49ff47cd7a3a2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "359c5d1eac914266be49ff47cd7a3a2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "359c5d1eac914266be49ff47cd7a3a2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45905 } } }
I20260929 18:02:20.528937 28862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 359c5d1eac914266be49ff47cd7a3a2c [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:20.531539 28870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:02:20.537169 28870 catalog_manager.cc:1347] Generated new cluster ID: 9a768bbd6d924bc29c4fe521e705869d
I20260929 18:02:20.537261 28870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:02:20.559212 28870 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:02:20.559600 28870 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:02:20.572090 28870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 359c5d1eac914266be49ff47cd7a3a2c: Generated new TSK 0
I20260929 18:02:20.573220 28870 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:02:20.578552 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 28783
2026-09-29T18:02:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1482 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:02:20 dist-test-slave-r7ph krb5kdc[28882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:02:20 dist-test-slave-r7ph krb5kdc[28882](info): set up 2 sockets
Sep 29 18:02:20 dist-test-slave-r7ph krb5kdc[28882](info): commencing operation
krb5kdc: starting...
W20260929 18:02:21.706588 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 18:02:21 dist-test-slave-r7ph krb5kdc[28882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:02:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:02:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:02:21.846629 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38099
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:21.973188 28895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:21.973582 28895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:21.973673 28895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:21.978164 28895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:21.978329 28895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:21.978367 28895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:21.978406 28895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:21.978440 28895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:21.984920 28895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38099
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:21.986937 28895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:21.988529 28895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:21.999341 28903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:21.999238 28901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:21.999238 28900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:21.999890 28895 server_base.cc:1034] running on GCE node
I20260929 18:02:22.000618 28895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:22.002187 28895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:22.003644 28895 hybrid_clock.cc:648] HybridClock initialized: now 1790704942003625 us; error 42 us; skew 500 ppm
Sep 29 18:02:22 dist-test-slave-r7ph krb5kdc[28882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:22.007694 28895 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:02:22.008874 28895 webserver.cc:460] Webserver started at http://127.0.0.1:41483/ using document root <none> and password file <none>
I20260929 18:02:22.009585 28895 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:22.009658 28895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:22.009888 28895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:22.012288 28895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "0efcc6cb8d9c4ae18721a74b579c50d7"
format_stamp: "Formatted at 2026-09-29 18:02:22 on dist-test-slave-r7ph"
server_key: "cd6e4aac28d2c64ddf67b0f22f050426"
server_key_iv: "60237e537d8b1cbd4a50cc7d0a7cdc42"
server_key_version: "encryptionkey@0"
I20260929 18:02:22.013347 28895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "0efcc6cb8d9c4ae18721a74b579c50d7"
format_stamp: "Formatted at 2026-09-29 18:02:22 on dist-test-slave-r7ph"
server_key: "cd6e4aac28d2c64ddf67b0f22f050426"
server_key_iv: "60237e537d8b1cbd4a50cc7d0a7cdc42"
server_key_version: "encryptionkey@0"
I20260929 18:02:22.017810 28895 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:02:22.020972 28910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:22.022805 28895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:02:22.022987 28895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "0efcc6cb8d9c4ae18721a74b579c50d7"
format_stamp: "Formatted at 2026-09-29 18:02:22 on dist-test-slave-r7ph"
server_key: "cd6e4aac28d2c64ddf67b0f22f050426"
server_key_iv: "60237e537d8b1cbd4a50cc7d0a7cdc42"
server_key_version: "encryptionkey@0"
I20260929 18:02:22.023142 28895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:22.044457 28895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:22.049642 28895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:22.049976 28895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:22.058151 28895 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38099
I20260929 18:02:22.058223 28962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38099 every 8 connection(s)
I20260929 18:02:22.059290 28895 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:02:22.063083 28963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:22.065205 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 28895
I20260929 18:02:22.065486 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:02:22.065788 7640 external_mini_cluster.cc:1428] Setting key e744608602f8ec67f54d9ad8052f2e0c
I20260929 18:02:22.071712 28963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0efcc6cb8d9c4ae18721a74b579c50d7: Bootstrap starting.
I20260929 18:02:22.074592 28963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0efcc6cb8d9c4ae18721a74b579c50d7: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:22.075443 28963 log.cc:824] T 00000000000000000000000000000000 P 0efcc6cb8d9c4ae18721a74b579c50d7: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:02:22 dist-test-slave-r7ph krb5kdc[28882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:02:22.078219 28963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0efcc6cb8d9c4ae18721a74b579c50d7: No bootstrap required, opened a new log
I20260929 18:02:22.082006 28963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0efcc6cb8d9c4ae18721a74b579c50d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0efcc6cb8d9c4ae18721a74b579c50d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38099 } }
I20260929 18:02:22.082423 28963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0efcc6cb8d9c4ae18721a74b579c50d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:22.082505 28963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0efcc6cb8d9c4ae18721a74b579c50d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0efcc6cb8d9c4ae18721a74b579c50d7, State: Initialized, Role: FOLLOWER
I20260929 18:02:22.083269 28966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:22.067444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:43187 (local address 127.0.0.1:38099)
0929 18:02:22.068214 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:22.068232 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:22.068274 (+ 42us) server_negotiation.cc:407] Connection header received
0929 18:02:22.069005 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:22.069026 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:22.069473 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:22.069898 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:22.070833 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:22.072132 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:22.073072 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:22.073603 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:22.073786 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:22.076979 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:22.076986 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:22.077002 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:22.077032 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:22.079167 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:22.079808 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:22.079814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:22.079820 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:22.079949 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:22.080316 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:22.080322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:22.080329 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:22.080955 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:22.081257 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:22.081868 (+ 611us) server_negotiation.cc:299] Negotiation successful
0929 18:02:22.082169 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":224,"threads_started":1}
I20260929 18:02:22.083249 28963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0efcc6cb8d9c4ae18721a74b579c50d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0efcc6cb8d9c4ae18721a74b579c50d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38099 } }
I20260929 18:02:22.083530 28963 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0efcc6cb8d9c4ae18721a74b579c50d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:02:22.083667 28963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0efcc6cb8d9c4ae18721a74b579c50d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:02:22.083817 28963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0efcc6cb8d9c4ae18721a74b579c50d7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:22.085234 28963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0efcc6cb8d9c4ae18721a74b579c50d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0efcc6cb8d9c4ae18721a74b579c50d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38099 } }
I20260929 18:02:22.085639 28963 leader_election.cc:302] T 00000000000000000000000000000000 P 0efcc6cb8d9c4ae18721a74b579c50d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0efcc6cb8d9c4ae18721a74b579c50d7; no voters:
I20260929 18:02:22.086086 28963 leader_election.cc:288] T 00000000000000000000000000000000 P 0efcc6cb8d9c4ae18721a74b579c50d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:02:22.086195 28968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0efcc6cb8d9c4ae18721a74b579c50d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:22.086557 28968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0efcc6cb8d9c4ae18721a74b579c50d7 [term 1 LEADER]: Becoming Leader. State: Replica: 0efcc6cb8d9c4ae18721a74b579c50d7, State: Running, Role: LEADER
I20260929 18:02:22.086946 28968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0efcc6cb8d9c4ae18721a74b579c50d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0efcc6cb8d9c4ae18721a74b579c50d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38099 } }
I20260929 18:02:22.087648 28963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0efcc6cb8d9c4ae18721a74b579c50d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:02:22.088115 7640 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:02:22.088335 28969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0efcc6cb8d9c4ae18721a74b579c50d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0efcc6cb8d9c4ae18721a74b579c50d7. Latest consensus state: current_term: 1 leader_uuid: "0efcc6cb8d9c4ae18721a74b579c50d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0efcc6cb8d9c4ae18721a74b579c50d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38099 } } }
I20260929 18:02:22.088449 28969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0efcc6cb8d9c4ae18721a74b579c50d7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:22.089129 28970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0efcc6cb8d9c4ae18721a74b579c50d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0efcc6cb8d9c4ae18721a74b579c50d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0efcc6cb8d9c4ae18721a74b579c50d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38099 } } }
I20260929 18:02:22.089470 28970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0efcc6cb8d9c4ae18721a74b579c50d7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:22.090206 28972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:02:22.095177 28972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:02:22.102308 28966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:22.090540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:46727 (local address 127.0.0.1:38099)
0929 18:02:22.090663 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:22.090667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:22.091171 (+ 504us) server_negotiation.cc:407] Connection header received
0929 18:02:22.091380 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:22.091383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:22.091439 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:22.091531 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:22.092482 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:22.093047 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:22.094244 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:22.094653 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:22.094731 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:22.096864 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:22.096869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:22.096872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:22.096906 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:22.098994 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:22.099765 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:22.099772 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:22.099777 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:22.099894 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:22.100789 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:22.100793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:22.100795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:22.101103 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:22.101228 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:22.101882 (+ 654us) server_negotiation.cc:299] Negotiation successful
0929 18:02:22.101996 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:02:22.105165 28972 catalog_manager.cc:1347] Generated new cluster ID: e5f93a3b2db0463a81623fb5166e3af3
I20260929 18:02:22.105262 28972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:02:22.117986 28972 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:02:22.119555 28972 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:02:22.125793 28972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0efcc6cb8d9c4ae18721a74b579c50d7: Generated new TSK 0
I20260929 18:02:22.127152 28972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:02:22.135872 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 28895
W20260929 18:02:22.141418 28880 connection.cc:535] client connection to 127.0.0.1:38099 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38099): Connection reset by peer (error 104)
2026-09-29T18:02:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1556 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:02:22 dist-test-slave-r7ph krb5kdc[28994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:02:22 dist-test-slave-r7ph krb5kdc[28994](info): set up 2 sockets
Sep 29 18:02:22 dist-test-slave-r7ph krb5kdc[28994](info): commencing operation
krb5kdc: starting...
W20260929 18:02:23.198858 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 18:02:23 dist-test-slave-r7ph krb5kdc[28994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:02:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:02:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:02:23.347373 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46879
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:23.468173 29007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:23.468492 29007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:23.468569 29007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:23.473243 29007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:23.473349 29007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:23.473371 29007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:23.473389 29007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:23.473407 29007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:23.479862 29007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46879
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:23.481151 29007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:23.482343 29007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:23.492147 29015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:23.492136 29012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:23.492128 29013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:23.492383 29007 server_base.cc:1034] running on GCE node
I20260929 18:02:23.493173 29007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:23.494656 29007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:23.495889 29007 hybrid_clock.cc:648] HybridClock initialized: now 1790704943495866 us; error 51 us; skew 500 ppm
Sep 29 18:02:23 dist-test-slave-r7ph krb5kdc[28994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:23.506894 29007 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:02:23.507997 29007 webserver.cc:460] Webserver started at http://127.0.0.1:44989/ using document root <none> and password file <none>
I20260929 18:02:23.508658 29007 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:23.508762 29007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:23.509021 29007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:23.511415 29007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "e897d4b3309c48a29e15f85082ff16bc"
format_stamp: "Formatted at 2026-09-29 18:02:23 on dist-test-slave-r7ph"
server_key: "5a0de0cb8e5d4b72bc204c84ac950b72"
server_key_iv: "0b8aae71a2718dc14a2da8035ed9a680"
server_key_version: "encryptionkey@0"
I20260929 18:02:23.512068 29007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "e897d4b3309c48a29e15f85082ff16bc"
format_stamp: "Formatted at 2026-09-29 18:02:23 on dist-test-slave-r7ph"
server_key: "5a0de0cb8e5d4b72bc204c84ac950b72"
server_key_iv: "0b8aae71a2718dc14a2da8035ed9a680"
server_key_version: "encryptionkey@0"
I20260929 18:02:23.516376 29007 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:02:23.519191 29022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:23.520565 29007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:02:23.520733 29007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "e897d4b3309c48a29e15f85082ff16bc"
format_stamp: "Formatted at 2026-09-29 18:02:23 on dist-test-slave-r7ph"
server_key: "5a0de0cb8e5d4b72bc204c84ac950b72"
server_key_iv: "0b8aae71a2718dc14a2da8035ed9a680"
server_key_version: "encryptionkey@0"
I20260929 18:02:23.520864 29007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:23.549009 29007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:23.553040 29007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:23.553295 29007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:23.561200 29007 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46879
I20260929 18:02:23.561213 29074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46879 every 8 connection(s)
I20260929 18:02:23.562296 29007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:02:23.565323 29075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:23.565622 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 29007
I20260929 18:02:23.565781 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:02:23.566103 7640 external_mini_cluster.cc:1428] Setting key 7027cae1a4776158960a66ae86bf2158
I20260929 18:02:23.573664 29075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e897d4b3309c48a29e15f85082ff16bc: Bootstrap starting.
I20260929 18:02:23.576761 29075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e897d4b3309c48a29e15f85082ff16bc: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:23.577755 29078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:23.567740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:58127 (local address 127.0.0.1:46879)
0929 18:02:23.568274 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:23.568295 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:23.568479 (+ 184us) server_negotiation.cc:407] Connection header received
0929 18:02:23.569376 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:23.569408 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:23.569959 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:23.570420 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:23.570868 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:23.572475 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:23.573521 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:23.574242 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:23.574434 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:23.575512 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:23.575519 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:23.575536 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:23.576073 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:23.576149 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:23.576181 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:23.576322 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:23.576544 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 18:02:23.576702 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":154,"threads_started":1}
I20260929 18:02:23.577713 29075 log.cc:824] T 00000000000000000000000000000000 P e897d4b3309c48a29e15f85082ff16bc: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:23.580204 29075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e897d4b3309c48a29e15f85082ff16bc: No bootstrap required, opened a new log
I20260929 18:02:23.581420 7640 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:02:23.585013 29075 raft_consensus.cc:348] T 00000000000000000000000000000000 P e897d4b3309c48a29e15f85082ff16bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e897d4b3309c48a29e15f85082ff16bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46879 } }
I20260929 18:02:23.585460 29075 raft_consensus.cc:374] T 00000000000000000000000000000000 P e897d4b3309c48a29e15f85082ff16bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:23.585566 29075 raft_consensus.cc:729] T 00000000000000000000000000000000 P e897d4b3309c48a29e15f85082ff16bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e897d4b3309c48a29e15f85082ff16bc, State: Initialized, Role: FOLLOWER
I20260929 18:02:23.586149 29075 consensus_queue.cc:260] T 00000000000000000000000000000000 P e897d4b3309c48a29e15f85082ff16bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e897d4b3309c48a29e15f85082ff16bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46879 } }
I20260929 18:02:23.586364 29075 raft_consensus.cc:388] T 00000000000000000000000000000000 P e897d4b3309c48a29e15f85082ff16bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:02:23.586493 29075 raft_consensus.cc:482] T 00000000000000000000000000000000 P e897d4b3309c48a29e15f85082ff16bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:02:23.586642 29075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e897d4b3309c48a29e15f85082ff16bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:23.588022 29075 raft_consensus.cc:504] T 00000000000000000000000000000000 P e897d4b3309c48a29e15f85082ff16bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e897d4b3309c48a29e15f85082ff16bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46879 } }
I20260929 18:02:23.588462 29075 leader_election.cc:302] T 00000000000000000000000000000000 P e897d4b3309c48a29e15f85082ff16bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e897d4b3309c48a29e15f85082ff16bc; no voters:
I20260929 18:02:23.588882 29075 leader_election.cc:288] T 00000000000000000000000000000000 P e897d4b3309c48a29e15f85082ff16bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:02:23.589043 29085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e897d4b3309c48a29e15f85082ff16bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:23.589329 29085 raft_consensus.cc:686] T 00000000000000000000000000000000 P e897d4b3309c48a29e15f85082ff16bc [term 1 LEADER]: Becoming Leader. State: Replica: e897d4b3309c48a29e15f85082ff16bc, State: Running, Role: LEADER
I20260929 18:02:23.589648 29078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:23.583799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:46119 (local address 127.0.0.1:46879)
0929 18:02:23.583974 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:23.583981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:23.584322 (+ 341us) server_negotiation.cc:407] Connection header received
0929 18:02:23.584534 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:23.584538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:23.584627 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:23.584795 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:23.585313 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:23.586181 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:23.587503 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:23.587898 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:23.588031 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:23.589037 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:23.589042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:23.589044 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:23.589060 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:23.589092 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:23.589106 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:23.589210 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:23.589460 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 18:02:23.589507 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:02:23.589748 29085 consensus_queue.cc:237] T 00000000000000000000000000000000 P e897d4b3309c48a29e15f85082ff16bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e897d4b3309c48a29e15f85082ff16bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46879 } }
I20260929 18:02:23.590624 29075 sys_catalog.cc:564] T 00000000000000000000000000000000 P e897d4b3309c48a29e15f85082ff16bc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:02:23.591606 29087 sys_catalog.cc:455] T 00000000000000000000000000000000 P e897d4b3309c48a29e15f85082ff16bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e897d4b3309c48a29e15f85082ff16bc. Latest consensus state: current_term: 1 leader_uuid: "e897d4b3309c48a29e15f85082ff16bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e897d4b3309c48a29e15f85082ff16bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46879 } } }
I20260929 18:02:23.591740 29087 sys_catalog.cc:458] T 00000000000000000000000000000000 P e897d4b3309c48a29e15f85082ff16bc [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:23.592141 29094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:02:23.592163 29086 sys_catalog.cc:455] T 00000000000000000000000000000000 P e897d4b3309c48a29e15f85082ff16bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e897d4b3309c48a29e15f85082ff16bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e897d4b3309c48a29e15f85082ff16bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46879 } } }
I20260929 18:02:23.592257 29086 sys_catalog.cc:458] T 00000000000000000000000000000000 P e897d4b3309c48a29e15f85082ff16bc [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:23.595825 29094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:02:23.602504 29094 catalog_manager.cc:1347] Generated new cluster ID: f8d3af6fb77c4350b0dba75a55e0c350
I20260929 18:02:23.602603 29094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:02:23.621371 29094 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:02:23.622653 29094 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:02:23.630915 29094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e897d4b3309c48a29e15f85082ff16bc: Generated new TSK 0
I20260929 18:02:23.631767 29094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:02:23.641507 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 29007
W20260929 18:02:23.647343 28992 connection.cc:535] client connection to 127.0.0.1:46879 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46879): Connection reset by peer (error 104)
2026-09-29T18:02:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1505 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:02:23 dist-test-slave-r7ph krb5kdc[29106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:02:23 dist-test-slave-r7ph krb5kdc[29106](info): set up 2 sockets
Sep 29 18:02:23 dist-test-slave-r7ph krb5kdc[29106](info): commencing operation
krb5kdc: starting...
W20260929 18:02:24.750782 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 18:02:24 dist-test-slave-r7ph krb5kdc[29106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:02:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:02:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:02:24.895967 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37249
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:25.024089 29119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:25.024438 29119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:25.024490 29119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:25.029395 29119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:25.029493 29119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:25.029515 29119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:25.029532 29119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:25.029547 29119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:25.035532 29119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37249
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704657566865-7640-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:25.036734 29119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:25.037842 29119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:25.048066 29127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:25.048116 29125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:25.048118 29119 server_base.cc:1034] running on GCE node
W20260929 18:02:25.048054 29124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:25.049116 29119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:25.050506 29119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:25.051735 29119 hybrid_clock.cc:648] HybridClock initialized: now 1790704945051677 us; error 77 us; skew 500 ppm
Sep 29 18:02:25 dist-test-slave-r7ph krb5kdc[29106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:25.054934 29119 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:02:25.056205 29119 webserver.cc:460] Webserver started at http://127.0.0.1:40629/ using document root <none> and password file <none>
I20260929 18:02:25.056879 29119 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:25.056962 29119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:25.057389 29119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:25.059719 29119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "023cb7eea7fe493b8fd03c55dbfc62c3"
format_stamp: "Formatted at 2026-09-29 18:02:25 on dist-test-slave-r7ph"
server_key: "bc311b461386e00f65d348647addf359"
server_key_iv: "2df0dd4005b901ce8f8f6c9d583d1f65"
server_key_version: "encryptionkey@0"
I20260929 18:02:25.060313 29119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "023cb7eea7fe493b8fd03c55dbfc62c3"
format_stamp: "Formatted at 2026-09-29 18:02:25 on dist-test-slave-r7ph"
server_key: "bc311b461386e00f65d348647addf359"
server_key_iv: "2df0dd4005b901ce8f8f6c9d583d1f65"
server_key_version: "encryptionkey@0"
I20260929 18:02:25.064666 29119 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:02:25.067734 29134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:25.069501 29119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:02:25.069685 29119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "023cb7eea7fe493b8fd03c55dbfc62c3"
format_stamp: "Formatted at 2026-09-29 18:02:25 on dist-test-slave-r7ph"
server_key: "bc311b461386e00f65d348647addf359"
server_key_iv: "2df0dd4005b901ce8f8f6c9d583d1f65"
server_key_version: "encryptionkey@0"
I20260929 18:02:25.069839 29119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:25.075393 29119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:25.079275 29119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:25.079470 29119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:25.087013 29119 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37249
I20260929 18:02:25.087070 29186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37249 every 8 connection(s)
I20260929 18:02:25.088129 29119 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:02:25.091434 29187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:25.093737 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 29119
I20260929 18:02:25.093873 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:02:25.094205 7640 external_mini_cluster.cc:1428] Setting key 961b316c39acca254ff9624e50f7d973
I20260929 18:02:25.098219 29187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 023cb7eea7fe493b8fd03c55dbfc62c3: Bootstrap starting.
I20260929 18:02:25.100060 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:25.095818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:48123 (local address 127.0.0.1:37249)
0929 18:02:25.096354 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:25.096374 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:25.096412 (+ 38us) server_negotiation.cc:407] Connection header received
0929 18:02:25.097167 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:25.097189 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:25.097505 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:25.097895 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:25.098771 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:25.098775 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:25.098788 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:25.098798 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:25.098852 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:25.098871 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:25.099041 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:25.099118 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 18:02:25.099296 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":208,"threads_started":1}
I20260929 18:02:25.100880 29187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 023cb7eea7fe493b8fd03c55dbfc62c3: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:25.101794 29187 log.cc:824] T 00000000000000000000000000000000 P 023cb7eea7fe493b8fd03c55dbfc62c3: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:25.103152 7640 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:02:25.104250 29187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 023cb7eea7fe493b8fd03c55dbfc62c3: No bootstrap required, opened a new log
I20260929 18:02:25.107033 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:25.105344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:54017 (local address 127.0.0.1:37249)
0929 18:02:25.105476 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:25.105481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:25.105697 (+ 216us) server_negotiation.cc:407] Connection header received
0929 18:02:25.105793 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:25.105796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:25.105834 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:25.105937 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:25.106643 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:25.106648 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:25.106650 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:25.106663 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:25.106692 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:25.106708 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:25.106819 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:25.106853 (+ 34us) server_negotiation.cc:299] Negotiation successful
0929 18:02:25.106892 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:02:25.108865 29187 raft_consensus.cc:348] T 00000000000000000000000000000000 P 023cb7eea7fe493b8fd03c55dbfc62c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "023cb7eea7fe493b8fd03c55dbfc62c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37249 } }
I20260929 18:02:25.109115 29187 raft_consensus.cc:374] T 00000000000000000000000000000000 P 023cb7eea7fe493b8fd03c55dbfc62c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:25.109150 29187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 023cb7eea7fe493b8fd03c55dbfc62c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 023cb7eea7fe493b8fd03c55dbfc62c3, State: Initialized, Role: FOLLOWER
I20260929 18:02:25.109565 29187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 023cb7eea7fe493b8fd03c55dbfc62c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "023cb7eea7fe493b8fd03c55dbfc62c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37249 } }
I20260929 18:02:25.109683 29187 raft_consensus.cc:388] T 00000000000000000000000000000000 P 023cb7eea7fe493b8fd03c55dbfc62c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:02:25.109723 29187 raft_consensus.cc:482] T 00000000000000000000000000000000 P 023cb7eea7fe493b8fd03c55dbfc62c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:02:25.109792 29187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 023cb7eea7fe493b8fd03c55dbfc62c3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:25.110826 29187 raft_consensus.cc:504] T 00000000000000000000000000000000 P 023cb7eea7fe493b8fd03c55dbfc62c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "023cb7eea7fe493b8fd03c55dbfc62c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37249 } }
I20260929 18:02:25.111151 29187 leader_election.cc:302] T 00000000000000000000000000000000 P 023cb7eea7fe493b8fd03c55dbfc62c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 023cb7eea7fe493b8fd03c55dbfc62c3; no voters:
I20260929 18:02:25.111436 29187 leader_election.cc:288] T 00000000000000000000000000000000 P 023cb7eea7fe493b8fd03c55dbfc62c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:02:25.111580 29197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 023cb7eea7fe493b8fd03c55dbfc62c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:25.111867 29197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 023cb7eea7fe493b8fd03c55dbfc62c3 [term 1 LEADER]: Becoming Leader. State: Replica: 023cb7eea7fe493b8fd03c55dbfc62c3, State: Running, Role: LEADER
I20260929 18:02:25.112226 29197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 023cb7eea7fe493b8fd03c55dbfc62c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "023cb7eea7fe493b8fd03c55dbfc62c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37249 } }
I20260929 18:02:25.112746 29187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 023cb7eea7fe493b8fd03c55dbfc62c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:02:25.114084 29198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 023cb7eea7fe493b8fd03c55dbfc62c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "023cb7eea7fe493b8fd03c55dbfc62c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "023cb7eea7fe493b8fd03c55dbfc62c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37249 } } }
I20260929 18:02:25.114234 29198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 023cb7eea7fe493b8fd03c55dbfc62c3 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:25.114677 29206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:02:25.114677 29199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 023cb7eea7fe493b8fd03c55dbfc62c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 023cb7eea7fe493b8fd03c55dbfc62c3. Latest consensus state: current_term: 1 leader_uuid: "023cb7eea7fe493b8fd03c55dbfc62c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "023cb7eea7fe493b8fd03c55dbfc62c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37249 } } }
I20260929 18:02:25.114868 29199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 023cb7eea7fe493b8fd03c55dbfc62c3 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:25.117717 29206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:02:25.123519 29206 catalog_manager.cc:1347] Generated new cluster ID: 817619708e2f4aa5a4199cca8470d6f0
I20260929 18:02:25.123610 29206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:02:25.152650 29206 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:02:25.153018 29206 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:02:25.159992 29206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 023cb7eea7fe493b8fd03c55dbfc62c3: Generated new TSK 0
I20260929 18:02:25.160812 29206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:02:25.165601 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 29119
2026-09-29T18:02:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1523 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:02:25 dist-test-slave-r7ph krb5kdc[29218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:02:25 dist-test-slave-r7ph krb5kdc[29218](info): set up 2 sockets
Sep 29 18:02:25 dist-test-slave-r7ph krb5kdc[29218](info): commencing operation
krb5kdc: starting...
W20260929 18:02:26.226677 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 18:02:26 dist-test-slave-r7ph krb5kdc[29218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:02:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:02:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 18:02:26.362326 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:43959
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:39391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:43959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:26.484724 29231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:26.485101 29231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:26.485189 29231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:26.489655 29231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:26.489763 29231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:26.489806 29231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:26.489872 29231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:26.489912 29231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:26.495939 29231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:39391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:43959
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:43959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:26.497207 29231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:26.498347 29231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:26.507944 29239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:26.508013 29236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:26.507944 29237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:26.508177 29231 server_base.cc:1034] running on GCE node
I20260929 18:02:26.509051 29231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:26.510466 29231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:26.511689 29231 hybrid_clock.cc:648] HybridClock initialized: now 1790704946511657 us; error 53 us; skew 500 ppm
Sep 29 18:02:26 dist-test-slave-r7ph krb5kdc[29218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704946, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:26.515033 29231 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 18:02:26.516201 29231 webserver.cc:460] Webserver started at http://127.7.118.62:45353/ using document root <none> and password file <none>
I20260929 18:02:26.516865 29231 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:26.516970 29231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:26.517233 29231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:26.519593 29231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "e2915e74f2f44fd494596e6bf98b8060"
format_stamp: "Formatted at 2026-09-29 18:02:26 on dist-test-slave-r7ph"
server_key: "2e7438575997d986f0bd3515d157a897"
server_key_iv: "64d3bca84d13f7df8d455f1c9db4508b"
server_key_version: "encryptionkey@0"
I20260929 18:02:26.520233 29231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "e2915e74f2f44fd494596e6bf98b8060"
format_stamp: "Formatted at 2026-09-29 18:02:26 on dist-test-slave-r7ph"
server_key: "2e7438575997d986f0bd3515d157a897"
server_key_iv: "64d3bca84d13f7df8d455f1c9db4508b"
server_key_version: "encryptionkey@0"
I20260929 18:02:26.524994 29231 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:02:26.528000 29246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:26.529734 29231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 18:02:26.529955 29231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "e2915e74f2f44fd494596e6bf98b8060"
format_stamp: "Formatted at 2026-09-29 18:02:26 on dist-test-slave-r7ph"
server_key: "2e7438575997d986f0bd3515d157a897"
server_key_iv: "64d3bca84d13f7df8d455f1c9db4508b"
server_key_version: "encryptionkey@0"
I20260929 18:02:26.530113 29231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:26.558466 29231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:26.562613 29231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:26.562877 29231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:26.571081 29231 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:43959
I20260929 18:02:26.571152 29298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:43959 every 8 connection(s)
I20260929 18:02:26.572363 29231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:02:26.575634 29299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:26.580298 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 29231
I20260929 18:02:26.580534 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:02:26.580837 7640 external_mini_cluster.cc:1428] Setting key 045e127d73bdf3acda971f3ffb7d82bd
I20260929 18:02:26.582168 29299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2915e74f2f44fd494596e6bf98b8060: Bootstrap starting.
I20260929 18:02:26.585235 29299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2915e74f2f44fd494596e6bf98b8060: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:26.586170 29299 log.cc:824] T 00000000000000000000000000000000 P e2915e74f2f44fd494596e6bf98b8060: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:26.588617 29299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2915e74f2f44fd494596e6bf98b8060: No bootstrap required, opened a new log
Sep 29 18:02:26 dist-test-slave-r7ph krb5kdc[29218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:02:26.593698 29299 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2915e74f2f44fd494596e6bf98b8060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2915e74f2f44fd494596e6bf98b8060" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 43959 } }
I20260929 18:02:26.594064 29299 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2915e74f2f44fd494596e6bf98b8060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:26.594125 29299 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2915e74f2f44fd494596e6bf98b8060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2915e74f2f44fd494596e6bf98b8060, State: Initialized, Role: FOLLOWER
I20260929 18:02:26.594732 29299 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2915e74f2f44fd494596e6bf98b8060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2915e74f2f44fd494596e6bf98b8060" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 43959 } }
I20260929 18:02:26.594926 29299 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2915e74f2f44fd494596e6bf98b8060 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:02:26.595005 29299 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2915e74f2f44fd494596e6bf98b8060 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:02:26.595117 29299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2915e74f2f44fd494596e6bf98b8060 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:26.596426 29299 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2915e74f2f44fd494596e6bf98b8060 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2915e74f2f44fd494596e6bf98b8060" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 43959 } }
I20260929 18:02:26.596895 29299 leader_election.cc:302] T 00000000000000000000000000000000 P e2915e74f2f44fd494596e6bf98b8060 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2915e74f2f44fd494596e6bf98b8060; no voters:
I20260929 18:02:26.597339 29299 leader_election.cc:288] T 00000000000000000000000000000000 P e2915e74f2f44fd494596e6bf98b8060 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:02:26.598510 29304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2915e74f2f44fd494596e6bf98b8060 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:26.598757 29304 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2915e74f2f44fd494596e6bf98b8060 [term 1 LEADER]: Becoming Leader. State: Replica: e2915e74f2f44fd494596e6bf98b8060, State: Running, Role: LEADER
I20260929 18:02:26.598981 29299 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2915e74f2f44fd494596e6bf98b8060 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:02:26.599229 29304 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2915e74f2f44fd494596e6bf98b8060 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2915e74f2f44fd494596e6bf98b8060" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 43959 } }
I20260929 18:02:26.599573 29302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:26.582510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52141 (local address 127.7.118.62:43959)
0929 18:02:26.582986 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:26.583007 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:26.583051 (+ 44us) server_negotiation.cc:407] Connection header received
0929 18:02:26.584048 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:26.584078 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:26.584572 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:26.584998 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:26.585970 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:26.587416 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:26.588494 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:26.589264 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:26.589439 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:26.592559 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:26.592570 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:26.592593 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:26.592636 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:26.595023 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:26.595816 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:26.595823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:26.595830 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:26.596022 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:26.596500 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:26.596504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:26.596506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:26.597205 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:26.597487 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:26.597999 (+ 512us) server_negotiation.cc:299] Negotiation successful
0929 18:02:26.598376 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":128,"threads_started":1}
I20260929 18:02:26.600754 29304 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2915e74f2f44fd494596e6bf98b8060 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2915e74f2f44fd494596e6bf98b8060. Latest consensus state: current_term: 1 leader_uuid: "e2915e74f2f44fd494596e6bf98b8060" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2915e74f2f44fd494596e6bf98b8060" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 43959 } } }
I20260929 18:02:26.600975 29304 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2915e74f2f44fd494596e6bf98b8060 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:26.601516 29319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:02:26.601922 29306 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2915e74f2f44fd494596e6bf98b8060 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2915e74f2f44fd494596e6bf98b8060" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2915e74f2f44fd494596e6bf98b8060" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 43959 } } }
I20260929 18:02:26.602075 29306 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2915e74f2f44fd494596e6bf98b8060 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:26.603770 7640 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:02:26.604905 29319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:02:26.612178 29319 catalog_manager.cc:1347] Generated new cluster ID: 83e843ae4fbd4352b61fbb31631316c0
I20260929 18:02:26.612270 29319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:02:26.615643 29302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:26.606391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38357 (local address 127.7.118.62:43959)
0929 18:02:26.606520 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:26.606525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:26.606771 (+ 246us) server_negotiation.cc:407] Connection header received
0929 18:02:26.606915 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:26.606918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:26.606973 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:26.607060 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:26.607870 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:26.608462 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:26.609336 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:26.609604 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:26.609719 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:26.611816 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:26.611819 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:26.611822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:26.611856 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:26.613515 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:26.614097 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:26.614101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:26.614103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:26.614192 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:26.614599 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:26.614602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:26.614604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:26.614795 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:26.614950 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:26.615234 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 18:02:26.615447 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:02:26.621978 29319 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:02:26.623612 29319 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:02:26.641302 29319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2915e74f2f44fd494596e6bf98b8060: Generated new TSK 0
I20260929 18:02:26.642306 29319 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:02:26.651719 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 29231
W20260929 18:02:26.657581 29216 connection.cc:535] client connection to 127.7.118.62:43959 recv error: Network error: failed to read from TLS socket (remote: 127.7.118.62:43959): Connection reset by peer (error 104)
2026-09-29T18:02:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1486 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:02:26 dist-test-slave-r7ph krb5kdc[29330](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:02:26 dist-test-slave-r7ph krb5kdc[29330](info): set up 2 sockets
Sep 29 18:02:26 dist-test-slave-r7ph krb5kdc[29330](info): commencing operation
krb5kdc: starting...
W20260929 18:02:27.726667 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 18:02:27 dist-test-slave-r7ph krb5kdc[29330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:02:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:02:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 18:02:27.861811 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:34389
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:34153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:34389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:27.991565 29343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:27.991918 29343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:27.991969 29343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:27.996894 29343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:27.996991 29343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:27.997013 29343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:27.997031 29343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:27.997047 29343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:28.003021 29343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:34153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:34389
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:34389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:28.004181 29343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:28.005261 29343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:28.015178 29348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:28.015208 29343 server_base.cc:1034] running on GCE node
W20260929 18:02:28.015393 29351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:28.015179 29349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:28.016018 29343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:28.017304 29343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:28.018491 29343 hybrid_clock.cc:648] HybridClock initialized: now 1790704948018474 us; error 34 us; skew 500 ppm
Sep 29 18:02:28 dist-test-slave-r7ph krb5kdc[29330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704948, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:28.021747 29343 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 18:02:28.022866 29343 webserver.cc:460] Webserver started at http://127.7.118.62:34779/ using document root <none> and password file <none>
I20260929 18:02:28.023475 29343 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:28.023543 29343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:28.023797 29343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:28.026034 29343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "1947fe2222224a01984acb52f8f3fc42"
format_stamp: "Formatted at 2026-09-29 18:02:28 on dist-test-slave-r7ph"
server_key: "af03df30163f37bf0147ad38d490d86d"
server_key_iv: "5c67afc64161a08d0e803bd63afbda97"
server_key_version: "encryptionkey@0"
I20260929 18:02:28.026703 29343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "1947fe2222224a01984acb52f8f3fc42"
format_stamp: "Formatted at 2026-09-29 18:02:28 on dist-test-slave-r7ph"
server_key: "af03df30163f37bf0147ad38d490d86d"
server_key_iv: "5c67afc64161a08d0e803bd63afbda97"
server_key_version: "encryptionkey@0"
I20260929 18:02:28.030858 29343 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:02:28.033685 29358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:28.035209 29343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:02:28.035410 29343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "1947fe2222224a01984acb52f8f3fc42"
format_stamp: "Formatted at 2026-09-29 18:02:28 on dist-test-slave-r7ph"
server_key: "af03df30163f37bf0147ad38d490d86d"
server_key_iv: "5c67afc64161a08d0e803bd63afbda97"
server_key_version: "encryptionkey@0"
I20260929 18:02:28.035552 29343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:28.058328 29343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:28.062041 29343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:28.062223 29343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:28.070286 29343 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:34389
I20260929 18:02:28.070312 29410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:34389 every 8 connection(s)
I20260929 18:02:28.071357 29343 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:02:28.074514 29411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:28.080030 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 29343
I20260929 18:02:28.080197 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:02:28.080489 7640 external_mini_cluster.cc:1428] Setting key 8529f51a3c151d952b6d8712febaf247
I20260929 18:02:28.080691 29411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1947fe2222224a01984acb52f8f3fc42: Bootstrap starting.
I20260929 18:02:28.083457 29411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1947fe2222224a01984acb52f8f3fc42: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:28.084275 29411 log.cc:824] T 00000000000000000000000000000000 P 1947fe2222224a01984acb52f8f3fc42: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:28.086380 29411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1947fe2222224a01984acb52f8f3fc42: No bootstrap required, opened a new log
I20260929 18:02:28.090241 29414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:28.081933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50253 (local address 127.7.118.62:34389)
0929 18:02:28.082526 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:28.082542 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:28.082582 (+ 40us) server_negotiation.cc:407] Connection header received
0929 18:02:28.083321 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:28.083341 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:28.083806 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:28.084187 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:28.084635 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:28.085632 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:28.086702 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:28.087217 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:28.087400 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:28.088372 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:28.088378 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:28.088393 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:28.088820 (+ 427us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:28.088880 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:28.088905 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:28.089053 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:28.089272 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 18:02:28.089425 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":202,"threads_started":1}
I20260929 18:02:28.090437 29411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1947fe2222224a01984acb52f8f3fc42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1947fe2222224a01984acb52f8f3fc42" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 34389 } }
I20260929 18:02:28.090730 29411 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1947fe2222224a01984acb52f8f3fc42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:28.090816 29411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1947fe2222224a01984acb52f8f3fc42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1947fe2222224a01984acb52f8f3fc42, State: Initialized, Role: FOLLOWER
I20260929 18:02:28.091262 29411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1947fe2222224a01984acb52f8f3fc42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1947fe2222224a01984acb52f8f3fc42" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 34389 } }
I20260929 18:02:28.091427 29411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1947fe2222224a01984acb52f8f3fc42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:02:28.091535 29411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1947fe2222224a01984acb52f8f3fc42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:02:28.091636 29411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1947fe2222224a01984acb52f8f3fc42 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:28.092867 29411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1947fe2222224a01984acb52f8f3fc42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1947fe2222224a01984acb52f8f3fc42" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 34389 } }
I20260929 18:02:28.093271 29411 leader_election.cc:302] T 00000000000000000000000000000000 P 1947fe2222224a01984acb52f8f3fc42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1947fe2222224a01984acb52f8f3fc42; no voters:
I20260929 18:02:28.093657 29411 leader_election.cc:288] T 00000000000000000000000000000000 P 1947fe2222224a01984acb52f8f3fc42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:02:28.093693 7640 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:02:28.093732 29416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1947fe2222224a01984acb52f8f3fc42 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:28.093923 29416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1947fe2222224a01984acb52f8f3fc42 [term 1 LEADER]: Becoming Leader. State: Replica: 1947fe2222224a01984acb52f8f3fc42, State: Running, Role: LEADER
I20260929 18:02:28.094223 29416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1947fe2222224a01984acb52f8f3fc42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1947fe2222224a01984acb52f8f3fc42" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 34389 } }
I20260929 18:02:28.095028 29411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1947fe2222224a01984acb52f8f3fc42 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:02:28.095928 29417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1947fe2222224a01984acb52f8f3fc42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1947fe2222224a01984acb52f8f3fc42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1947fe2222224a01984acb52f8f3fc42" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 34389 } } }
I20260929 18:02:28.096066 29417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1947fe2222224a01984acb52f8f3fc42 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:28.096632 29418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1947fe2222224a01984acb52f8f3fc42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1947fe2222224a01984acb52f8f3fc42. Latest consensus state: current_term: 1 leader_uuid: "1947fe2222224a01984acb52f8f3fc42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1947fe2222224a01984acb52f8f3fc42" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 34389 } } }
I20260929 18:02:28.096745 29418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1947fe2222224a01984acb52f8f3fc42 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:28.097167 29430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:02:28.101095 29430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:02:28.101476 29414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:28.096029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33981 (local address 127.7.118.62:34389)
0929 18:02:28.096470 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:28.096475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:28.096489 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:02:28.096640 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:28.096645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:28.096699 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:28.096770 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:28.097265 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:28.098086 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:28.099281 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:28.099597 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:28.099672 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:28.100883 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:28.100887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:28.100889 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:28.100904 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:28.100935 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:28.100950 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:28.101014 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:28.101304 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 18:02:28.101345 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351}
I20260929 18:02:28.107149 29430 catalog_manager.cc:1347] Generated new cluster ID: 4cca2b28173f425480c3b35c5a36955e
I20260929 18:02:28.107241 29430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:02:28.120990 29430 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:02:28.122448 29430 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:02:28.128203 29430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1947fe2222224a01984acb52f8f3fc42: Generated new TSK 0
I20260929 18:02:28.129014 29430 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:02:28.140972 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 29343
W20260929 18:02:28.146456 29328 connection.cc:535] client connection to 127.7.118.62:34389 recv error: Network error: failed to read from TLS socket (remote: 127.7.118.62:34389): Connection reset by peer (error 104)
2026-09-29T18:02:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1488 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:02:28 dist-test-slave-r7ph krb5kdc[29442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:02:28 dist-test-slave-r7ph krb5kdc[29442](info): set up 2 sockets
Sep 29 18:02:28 dist-test-slave-r7ph krb5kdc[29442](info): commencing operation
krb5kdc: starting...
W20260929 18:02:29.219090 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 18:02:29 dist-test-slave-r7ph krb5kdc[29442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:02:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:02:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 18:02:29.358330 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:35607
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:38611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:35607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:29.483286 29455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:29.483651 29455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:29.483700 29455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:29.488444 29455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:29.488543 29455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:29.488564 29455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:29.488581 29455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:29.488597 29455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:29.495137 29455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:38611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:35607
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:35607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:29.496739 29455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:29.498325 29455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:29.509739 29463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:29.509780 29461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:29.509780 29460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:29.509881 29455 server_base.cc:1034] running on GCE node
I20260929 18:02:29.510896 29455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:29.512189 29455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:29.513434 29455 hybrid_clock.cc:648] HybridClock initialized: now 1790704949513416 us; error 37 us; skew 500 ppm
Sep 29 18:02:29 dist-test-slave-r7ph krb5kdc[29442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704949, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:29.523770 29455 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 18:02:29.524953 29455 webserver.cc:460] Webserver started at http://127.7.118.62:33323/ using document root <none> and password file <none>
I20260929 18:02:29.525689 29455 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:29.525800 29455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:29.526073 29455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:29.528487 29455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "797dc31da7fc4ea8b81404d760ccffe8"
format_stamp: "Formatted at 2026-09-29 18:02:29 on dist-test-slave-r7ph"
server_key: "6c2a45e1eb8cc32ec5b97ad6e5400b68"
server_key_iv: "6565641d955530fcf003f05f56721f92"
server_key_version: "encryptionkey@0"
I20260929 18:02:29.529126 29455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "797dc31da7fc4ea8b81404d760ccffe8"
format_stamp: "Formatted at 2026-09-29 18:02:29 on dist-test-slave-r7ph"
server_key: "6c2a45e1eb8cc32ec5b97ad6e5400b68"
server_key_iv: "6565641d955530fcf003f05f56721f92"
server_key_version: "encryptionkey@0"
I20260929 18:02:29.533452 29455 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:02:29.536715 29470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:29.538122 29455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:02:29.538376 29455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "797dc31da7fc4ea8b81404d760ccffe8"
format_stamp: "Formatted at 2026-09-29 18:02:29 on dist-test-slave-r7ph"
server_key: "6c2a45e1eb8cc32ec5b97ad6e5400b68"
server_key_iv: "6565641d955530fcf003f05f56721f92"
server_key_version: "encryptionkey@0"
I20260929 18:02:29.538529 29455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:29.543547 29455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:29.547240 29455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:29.547495 29455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:29.555374 29455 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:35607
I20260929 18:02:29.555387 29522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:35607 every 8 connection(s)
I20260929 18:02:29.556473 29455 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:02:29.559430 29523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:29.565366 29523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 797dc31da7fc4ea8b81404d760ccffe8: Bootstrap starting.
I20260929 18:02:29.566066 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 29455
I20260929 18:02:29.566234 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:02:29.566547 7640 external_mini_cluster.cc:1428] Setting key 46006fcbc1a6e904ef9350fccf6a2142
I20260929 18:02:29.568507 29523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 797dc31da7fc4ea8b81404d760ccffe8: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:29.569360 29523 log.cc:824] T 00000000000000000000000000000000 P 797dc31da7fc4ea8b81404d760ccffe8: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:29.572072 29523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 797dc31da7fc4ea8b81404d760ccffe8: No bootstrap required, opened a new log
I20260929 18:02:29.572803 29526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:29.567917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50081 (local address 127.7.118.62:35607)
0929 18:02:29.568440 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:29.568456 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:29.568495 (+ 39us) server_negotiation.cc:407] Connection header received
0929 18:02:29.569364 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:29.569385 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:29.569820 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:29.570314 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:29.571350 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:29.571355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:29.571369 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:29.571381 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:29.571447 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:29.571468 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:29.571569 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:29.571745 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 18:02:29.571942 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":137,"threads_started":1}
I20260929 18:02:29.576347 7640 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:02:29.576705 29523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 797dc31da7fc4ea8b81404d760ccffe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797dc31da7fc4ea8b81404d760ccffe8" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35607 } }
I20260929 18:02:29.576992 29523 raft_consensus.cc:374] T 00000000000000000000000000000000 P 797dc31da7fc4ea8b81404d760ccffe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:29.577069 29523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 797dc31da7fc4ea8b81404d760ccffe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 797dc31da7fc4ea8b81404d760ccffe8, State: Initialized, Role: FOLLOWER
I20260929 18:02:29.577575 29523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 797dc31da7fc4ea8b81404d760ccffe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797dc31da7fc4ea8b81404d760ccffe8" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35607 } }
I20260929 18:02:29.577736 29523 raft_consensus.cc:388] T 00000000000000000000000000000000 P 797dc31da7fc4ea8b81404d760ccffe8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:02:29.577831 29523 raft_consensus.cc:482] T 00000000000000000000000000000000 P 797dc31da7fc4ea8b81404d760ccffe8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:02:29.577975 29523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 797dc31da7fc4ea8b81404d760ccffe8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:29.579365 29523 raft_consensus.cc:504] T 00000000000000000000000000000000 P 797dc31da7fc4ea8b81404d760ccffe8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797dc31da7fc4ea8b81404d760ccffe8" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35607 } }
I20260929 18:02:29.579865 29523 leader_election.cc:302] T 00000000000000000000000000000000 P 797dc31da7fc4ea8b81404d760ccffe8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 797dc31da7fc4ea8b81404d760ccffe8; no voters:
I20260929 18:02:29.580310 29523 leader_election.cc:288] T 00000000000000000000000000000000 P 797dc31da7fc4ea8b81404d760ccffe8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:02:29.580390 29533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 797dc31da7fc4ea8b81404d760ccffe8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:29.580529 29526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:29.578674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49469 (local address 127.7.118.62:35607)
0929 18:02:29.578845 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:29.578850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:29.579013 (+ 163us) server_negotiation.cc:407] Connection header received
0929 18:02:29.579187 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:29.579190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:29.579233 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:29.579328 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:29.580127 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:29.580131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:29.580134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:29.580146 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:29.580190 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:29.580202 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:29.580297 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:29.580367 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 18:02:29.580410 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:02:29.580762 29533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 797dc31da7fc4ea8b81404d760ccffe8 [term 1 LEADER]: Becoming Leader. State: Replica: 797dc31da7fc4ea8b81404d760ccffe8, State: Running, Role: LEADER
I20260929 18:02:29.581112 29533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 797dc31da7fc4ea8b81404d760ccffe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797dc31da7fc4ea8b81404d760ccffe8" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35607 } }
I20260929 18:02:29.581861 29523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 797dc31da7fc4ea8b81404d760ccffe8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:02:29.582479 29535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 797dc31da7fc4ea8b81404d760ccffe8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 797dc31da7fc4ea8b81404d760ccffe8. Latest consensus state: current_term: 1 leader_uuid: "797dc31da7fc4ea8b81404d760ccffe8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797dc31da7fc4ea8b81404d760ccffe8" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35607 } } }
I20260929 18:02:29.582640 29535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 797dc31da7fc4ea8b81404d760ccffe8 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:29.582928 29534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 797dc31da7fc4ea8b81404d760ccffe8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "797dc31da7fc4ea8b81404d760ccffe8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797dc31da7fc4ea8b81404d760ccffe8" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35607 } } }
I20260929 18:02:29.583099 29534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 797dc31da7fc4ea8b81404d760ccffe8 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:29.584524 29539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:02:29.587675 29539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:02:29.594815 29539 catalog_manager.cc:1347] Generated new cluster ID: 17de261c5ee64ff48ce20307dff84f69
I20260929 18:02:29.594925 29539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:02:29.606666 29539 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:02:29.607089 29539 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:02:29.616217 29539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 797dc31da7fc4ea8b81404d760ccffe8: Generated new TSK 0
I20260929 18:02:29.617206 29539 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:02:29.628391 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 29455
2026-09-29T18:02:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1486 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:02:29 dist-test-slave-r7ph krb5kdc[29554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:02:29 dist-test-slave-r7ph krb5kdc[29554](info): set up 2 sockets
Sep 29 18:02:29 dist-test-slave-r7ph krb5kdc[29554](info): commencing operation
krb5kdc: starting...
W20260929 18:02:30.726629 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 18:02:30 dist-test-slave-r7ph krb5kdc[29554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:02:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:02:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 18:02:30.862074 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:33733
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:42565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:33733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:30.979069 29567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:30.979408 29567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:30.979453 29567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:30.983776 29567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:30.983871 29567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:30.983892 29567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:30.983911 29567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:30.983927 29567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:30.990123 29567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:42565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:33733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:33733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:30.991310 29567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:30.992357 29567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:02:31.002055 29567 server_base.cc:1034] running on GCE node
W20260929 18:02:31.001901 29573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:31.001925 29572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:31.001901 29575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:31.003023 29567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:31.004298 29567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:31.005586 29567 hybrid_clock.cc:648] HybridClock initialized: now 1790704951005521 us; error 40 us; skew 500 ppm
Sep 29 18:02:31 dist-test-slave-r7ph krb5kdc[29554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704951, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:31.008999 29567 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 18:02:31.010226 29567 webserver.cc:460] Webserver started at http://127.7.118.62:37283/ using document root <none> and password file <none>
I20260929 18:02:31.010950 29567 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:31.011025 29567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:31.011281 29567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:31.013633 29567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "3f0cc0b1ae314665b84360f8cdefc541"
format_stamp: "Formatted at 2026-09-29 18:02:31 on dist-test-slave-r7ph"
server_key: "43dfd0c302b12b0ed446fd89296be652"
server_key_iv: "16f24856f6e80d51370de4fd9f08acf8"
server_key_version: "encryptionkey@0"
I20260929 18:02:31.014371 29567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "3f0cc0b1ae314665b84360f8cdefc541"
format_stamp: "Formatted at 2026-09-29 18:02:31 on dist-test-slave-r7ph"
server_key: "43dfd0c302b12b0ed446fd89296be652"
server_key_iv: "16f24856f6e80d51370de4fd9f08acf8"
server_key_version: "encryptionkey@0"
I20260929 18:02:31.018800 29567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 18:02:31.021965 29582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:31.023608 29567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:02:31.023785 29567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "3f0cc0b1ae314665b84360f8cdefc541"
format_stamp: "Formatted at 2026-09-29 18:02:31 on dist-test-slave-r7ph"
server_key: "43dfd0c302b12b0ed446fd89296be652"
server_key_iv: "16f24856f6e80d51370de4fd9f08acf8"
server_key_version: "encryptionkey@0"
I20260929 18:02:31.023922 29567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:31.038121 29567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:31.042011 29567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:31.042326 29567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:31.051352 29567 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:33733
I20260929 18:02:31.051381 29634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:33733 every 8 connection(s)
I20260929 18:02:31.052505 29567 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:02:31.055505 29635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:31.059556 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 29567
I20260929 18:02:31.059691 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:02:31.060001 7640 external_mini_cluster.cc:1428] Setting key 69f5fae9289b0124fe6cd7a30341cc78
I20260929 18:02:31.062390 29635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f0cc0b1ae314665b84360f8cdefc541: Bootstrap starting.
I20260929 18:02:31.065439 29635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f0cc0b1ae314665b84360f8cdefc541: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:31.066434 29635 log.cc:824] T 00000000000000000000000000000000 P 3f0cc0b1ae314665b84360f8cdefc541: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:31.069218 29635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f0cc0b1ae314665b84360f8cdefc541: No bootstrap required, opened a new log
Sep 29 18:02:31 dist-test-slave-r7ph krb5kdc[29554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:02:31.073880 29635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f0cc0b1ae314665b84360f8cdefc541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f0cc0b1ae314665b84360f8cdefc541" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33733 } }
I20260929 18:02:31.074219 29635 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f0cc0b1ae314665b84360f8cdefc541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:31.074304 29635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f0cc0b1ae314665b84360f8cdefc541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f0cc0b1ae314665b84360f8cdefc541, State: Initialized, Role: FOLLOWER
I20260929 18:02:31.074822 29635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f0cc0b1ae314665b84360f8cdefc541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f0cc0b1ae314665b84360f8cdefc541" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33733 } }
I20260929 18:02:31.074994 29635 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f0cc0b1ae314665b84360f8cdefc541 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:02:31.075069 29635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f0cc0b1ae314665b84360f8cdefc541 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:02:31.075212 29635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f0cc0b1ae314665b84360f8cdefc541 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:31.076517 29635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f0cc0b1ae314665b84360f8cdefc541 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f0cc0b1ae314665b84360f8cdefc541" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33733 } }
I20260929 18:02:31.076948 29635 leader_election.cc:302] T 00000000000000000000000000000000 P 3f0cc0b1ae314665b84360f8cdefc541 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f0cc0b1ae314665b84360f8cdefc541; no voters:
I20260929 18:02:31.077301 29635 leader_election.cc:288] T 00000000000000000000000000000000 P 3f0cc0b1ae314665b84360f8cdefc541 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:02:31.077445 29640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f0cc0b1ae314665b84360f8cdefc541 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:31.077690 29640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f0cc0b1ae314665b84360f8cdefc541 [term 1 LEADER]: Becoming Leader. State: Replica: 3f0cc0b1ae314665b84360f8cdefc541, State: Running, Role: LEADER
I20260929 18:02:31.077983 29638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:31.061625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53819 (local address 127.7.118.62:33733)
0929 18:02:31.062105 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:31.062130 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:31.062175 (+ 45us) server_negotiation.cc:407] Connection header received
0929 18:02:31.063087 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:31.063112 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:31.063594 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:31.064025 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:31.064813 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:31.066405 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:31.067656 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:31.068412 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:31.068580 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:31.071675 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:31.071685 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:31.071717 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:31.071759 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:31.074158 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:31.074826 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:31.074833 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:31.074842 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:31.074993 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:31.075448 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:31.075455 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:31.075459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:31.075996 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:31.076216 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:31.076552 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 18:02:31.076882 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":144,"threads_started":1}
I20260929 18:02:31.078087 29640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f0cc0b1ae314665b84360f8cdefc541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f0cc0b1ae314665b84360f8cdefc541" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33733 } }
I20260929 18:02:31.078701 29635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f0cc0b1ae314665b84360f8cdefc541 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:02:31.079941 29641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f0cc0b1ae314665b84360f8cdefc541 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f0cc0b1ae314665b84360f8cdefc541" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f0cc0b1ae314665b84360f8cdefc541" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33733 } } }
I20260929 18:02:31.080071 29641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f0cc0b1ae314665b84360f8cdefc541 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:31.080432 29642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f0cc0b1ae314665b84360f8cdefc541 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f0cc0b1ae314665b84360f8cdefc541. Latest consensus state: current_term: 1 leader_uuid: "3f0cc0b1ae314665b84360f8cdefc541" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f0cc0b1ae314665b84360f8cdefc541" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 33733 } } }
I20260929 18:02:31.081096 29642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f0cc0b1ae314665b84360f8cdefc541 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:31.081197 29644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:02:31.081684 7640 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:02:31.084589 29644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:02:31.090859 29644 catalog_manager.cc:1347] Generated new cluster ID: 38214360ad9d473583d5bd159698f2c7
I20260929 18:02:31.090939 29644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:02:31.098287 29638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:31.086856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33709 (local address 127.7.118.62:33733)
0929 18:02:31.086977 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:31.086981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:31.089010 (+ 2029us) server_negotiation.cc:407] Connection header received
0929 18:02:31.089214 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:31.089218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:31.089275 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:31.089375 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:31.090338 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:31.091199 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:31.092393 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:31.092833 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:31.092956 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:31.094641 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:31.094645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:31.094648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:31.094680 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:31.096461 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:31.096960 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:31.096964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:31.096966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:31.097059 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:31.097344 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:31.097348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:31.097349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:31.097593 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:31.097730 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:31.097962 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 18:02:31.098092 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:02:31.104616 29644 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:02:31.105994 29644 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:02:31.132982 29644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f0cc0b1ae314665b84360f8cdefc541: Generated new TSK 0
I20260929 18:02:31.133956 29644 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:02:31.139533 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 29567
2026-09-29T18:02:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1511 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:02:31 dist-test-slave-r7ph krb5kdc[29666](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:02:31 dist-test-slave-r7ph krb5kdc[29666](info): set up 2 sockets
Sep 29 18:02:31 dist-test-slave-r7ph krb5kdc[29666](info): commencing operation
krb5kdc: starting...
W20260929 18:02:32.215167 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 18:02:32 dist-test-slave-r7ph krb5kdc[29666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:02:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:02:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 18:02:32.350240 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:34641
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:38643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:34641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:32.485257 29679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:32.485644 29679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:32.485697 29679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:32.490417 29679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:32.490506 29679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:32.490527 29679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:32.490545 29679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:32.490561 29679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:32.496431 29679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:38643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:34641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:34641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:32.497668 29679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:32.498790 29679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:02:32.508636 29679 server_base.cc:1034] running on GCE node
W20260929 18:02:32.508571 29684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:32.508599 29685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:32.508646 29687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:32.509567 29679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:32.510900 29679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:32.512116 29679 hybrid_clock.cc:648] HybridClock initialized: now 1790704952512090 us; error 47 us; skew 500 ppm
Sep 29 18:02:32 dist-test-slave-r7ph krb5kdc[29666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704952, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:32.515308 29679 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 18:02:32.516291 29679 webserver.cc:460] Webserver started at http://127.7.118.62:43699/ using document root <none> and password file <none>
I20260929 18:02:32.516871 29679 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:32.516934 29679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:32.517110 29679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:32.519384 29679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "90d5984cf4f84351a681d89cd7a4ec00"
format_stamp: "Formatted at 2026-09-29 18:02:32 on dist-test-slave-r7ph"
server_key: "fe3dea88fe4f8d9b89131511eb4be8ae"
server_key_iv: "74399453ec38d38f438c6d6ab929b917"
server_key_version: "encryptionkey@0"
I20260929 18:02:32.519939 29679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "90d5984cf4f84351a681d89cd7a4ec00"
format_stamp: "Formatted at 2026-09-29 18:02:32 on dist-test-slave-r7ph"
server_key: "fe3dea88fe4f8d9b89131511eb4be8ae"
server_key_iv: "74399453ec38d38f438c6d6ab929b917"
server_key_version: "encryptionkey@0"
I20260929 18:02:32.524039 29679 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:02:32.527112 29694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:32.528700 29679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 18:02:32.528877 29679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "90d5984cf4f84351a681d89cd7a4ec00"
format_stamp: "Formatted at 2026-09-29 18:02:32 on dist-test-slave-r7ph"
server_key: "fe3dea88fe4f8d9b89131511eb4be8ae"
server_key_iv: "74399453ec38d38f438c6d6ab929b917"
server_key_version: "encryptionkey@0"
I20260929 18:02:32.529030 29679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:32.550045 29679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:32.553814 29679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:32.554069 29679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:32.562084 29679 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:34641
I20260929 18:02:32.562103 29746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:34641 every 8 connection(s)
I20260929 18:02:32.563208 29679 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:02:32.566437 29747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:32.568534 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 29679
I20260929 18:02:32.568698 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:02:32.569010 7640 external_mini_cluster.cc:1428] Setting key d417c0a2d465a7b1a3393f3bc161c284
I20260929 18:02:32.573575 29747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90d5984cf4f84351a681d89cd7a4ec00: Bootstrap starting.
I20260929 18:02:32.576089 29747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90d5984cf4f84351a681d89cd7a4ec00: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:32.576957 29747 log.cc:824] T 00000000000000000000000000000000 P 90d5984cf4f84351a681d89cd7a4ec00: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:32.578675 29750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:32.570618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36141 (local address 127.7.118.62:34641)
0929 18:02:32.571122 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:32.571143 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:32.571183 (+ 40us) server_negotiation.cc:407] Connection header received
0929 18:02:32.571888 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:32.571908 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:32.572287 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:32.572681 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:32.573115 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:32.574169 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:32.575093 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:32.575597 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:32.575793 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:32.576738 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:32.576749 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:32.576765 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:32.577240 (+ 475us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:32.577301 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:32.577326 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:32.577439 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:32.577608 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 18:02:32.577842 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":141,"threads_started":1}
I20260929 18:02:32.579190 29747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90d5984cf4f84351a681d89cd7a4ec00: No bootstrap required, opened a new log
I20260929 18:02:32.582028 7640 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:02:32.583606 29747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90d5984cf4f84351a681d89cd7a4ec00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d5984cf4f84351a681d89cd7a4ec00" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 34641 } }
I20260929 18:02:32.583904 29747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90d5984cf4f84351a681d89cd7a4ec00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:32.583973 29747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90d5984cf4f84351a681d89cd7a4ec00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d5984cf4f84351a681d89cd7a4ec00, State: Initialized, Role: FOLLOWER
I20260929 18:02:32.584553 29747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90d5984cf4f84351a681d89cd7a4ec00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d5984cf4f84351a681d89cd7a4ec00" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 34641 } }
I20260929 18:02:32.584739 29747 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90d5984cf4f84351a681d89cd7a4ec00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:02:32.584842 29747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90d5984cf4f84351a681d89cd7a4ec00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:02:32.584975 29747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90d5984cf4f84351a681d89cd7a4ec00 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:32.586289 29747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90d5984cf4f84351a681d89cd7a4ec00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d5984cf4f84351a681d89cd7a4ec00" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 34641 } }
I20260929 18:02:32.586689 29747 leader_election.cc:302] T 00000000000000000000000000000000 P 90d5984cf4f84351a681d89cd7a4ec00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90d5984cf4f84351a681d89cd7a4ec00; no voters:
I20260929 18:02:32.587061 29747 leader_election.cc:288] T 00000000000000000000000000000000 P 90d5984cf4f84351a681d89cd7a4ec00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:02:32.587193 29757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90d5984cf4f84351a681d89cd7a4ec00 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:32.587457 29757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90d5984cf4f84351a681d89cd7a4ec00 [term 1 LEADER]: Becoming Leader. State: Replica: 90d5984cf4f84351a681d89cd7a4ec00, State: Running, Role: LEADER
I20260929 18:02:32.587888 29757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90d5984cf4f84351a681d89cd7a4ec00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d5984cf4f84351a681d89cd7a4ec00" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 34641 } }
I20260929 18:02:32.588577 29747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90d5984cf4f84351a681d89cd7a4ec00 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:02:32.589493 29750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:32.584399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42279 (local address 127.7.118.62:34641)
0929 18:02:32.584532 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:32.584535 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:32.584770 (+ 235us) server_negotiation.cc:407] Connection header received
0929 18:02:32.584920 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:32.584924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:32.584982 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:32.585078 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:32.585387 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:32.586139 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:32.587224 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:32.587561 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:32.587675 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:32.588756 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:32.588760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:32.588763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:32.588778 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:32.588810 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:32.588822 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:32.588884 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:32.589250 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 18:02:32.589311 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:02:32.589795 29758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90d5984cf4f84351a681d89cd7a4ec00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90d5984cf4f84351a681d89cd7a4ec00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d5984cf4f84351a681d89cd7a4ec00" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 34641 } } }
I20260929 18:02:32.589965 29758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90d5984cf4f84351a681d89cd7a4ec00 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:32.590461 29759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90d5984cf4f84351a681d89cd7a4ec00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90d5984cf4f84351a681d89cd7a4ec00. Latest consensus state: current_term: 1 leader_uuid: "90d5984cf4f84351a681d89cd7a4ec00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d5984cf4f84351a681d89cd7a4ec00" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 34641 } } }
I20260929 18:02:32.590564 29759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90d5984cf4f84351a681d89cd7a4ec00 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:32.590698 29766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:02:32.593740 29766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:02:32.599411 29766 catalog_manager.cc:1347] Generated new cluster ID: 8eceb781d89c4f16ab46ba0473ef172f
I20260929 18:02:32.599510 29766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:02:32.621066 29766 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:02:32.622134 29766 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:02:32.633123 29766 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90d5984cf4f84351a681d89cd7a4ec00: Generated new TSK 0
I20260929 18:02:32.634017 29766 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:02:32.643061 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 29679
2026-09-29T18:02:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1503 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:02:32 dist-test-slave-r7ph krb5kdc[29778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:02:32 dist-test-slave-r7ph krb5kdc[29778](info): set up 2 sockets
Sep 29 18:02:32 dist-test-slave-r7ph krb5kdc[29778](info): commencing operation
krb5kdc: starting...
W20260929 18:02:33.710700 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 18:02:33 dist-test-slave-r7ph krb5kdc[29778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:02:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:02:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 18:02:33.850704 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:35927
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:46623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:35927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:33.967724 29791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:33.968045 29791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:33.968089 29791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:33.972508 29791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:33.972604 29791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:33.972630 29791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:33.972648 29791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:33.972663 29791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:33.978478 29791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:46623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:35927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:35927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:33.979705 29791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:33.980826 29791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:33.990617 29796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:33.990643 29799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:33.990687 29797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:33.990633 29791 server_base.cc:1034] running on GCE node
I20260929 18:02:33.991659 29791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:33.993026 29791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:33.994343 29791 hybrid_clock.cc:648] HybridClock initialized: now 1790704953994324 us; error 37 us; skew 500 ppm
Sep 29 18:02:33 dist-test-slave-r7ph krb5kdc[29778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704953, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:33.997678 29791 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 18:02:33.998776 29791 webserver.cc:460] Webserver started at http://127.7.118.62:40035/ using document root <none> and password file <none>
I20260929 18:02:33.999382 29791 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:33.999445 29791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:33.999639 29791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:34.001993 29791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "2eed152a1c034e8d877e0a0424118bed"
format_stamp: "Formatted at 2026-09-29 18:02:33 on dist-test-slave-r7ph"
server_key: "6252c12091d8dbc90b0277d0111fc2ff"
server_key_iv: "743fcbb458aaeb57c35c0b9d5b08018f"
server_key_version: "encryptionkey@0"
I20260929 18:02:34.002718 29791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "2eed152a1c034e8d877e0a0424118bed"
format_stamp: "Formatted at 2026-09-29 18:02:33 on dist-test-slave-r7ph"
server_key: "6252c12091d8dbc90b0277d0111fc2ff"
server_key_iv: "743fcbb458aaeb57c35c0b9d5b08018f"
server_key_version: "encryptionkey@0"
I20260929 18:02:34.007082 29791 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:02:34.009825 29806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:34.011377 29791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:02:34.011514 29791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "2eed152a1c034e8d877e0a0424118bed"
format_stamp: "Formatted at 2026-09-29 18:02:33 on dist-test-slave-r7ph"
server_key: "6252c12091d8dbc90b0277d0111fc2ff"
server_key_iv: "743fcbb458aaeb57c35c0b9d5b08018f"
server_key_version: "encryptionkey@0"
I20260929 18:02:34.011629 29791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:34.016610 29791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:34.020133 29791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:34.020300 29791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:34.028198 29791 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:35927
I20260929 18:02:34.028218 29858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:35927 every 8 connection(s)
I20260929 18:02:34.029440 29791 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:02:34.032616 29859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:34.037832 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 29791
I20260929 18:02:34.038012 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:02:34.038321 7640 external_mini_cluster.cc:1428] Setting key 4878eb0abbf2f1e321285dfa3b35e8d5
I20260929 18:02:34.039150 29859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eed152a1c034e8d877e0a0424118bed: Bootstrap starting.
I20260929 18:02:34.041917 29859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2eed152a1c034e8d877e0a0424118bed: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:34.042987 29859 log.cc:824] T 00000000000000000000000000000000 P 2eed152a1c034e8d877e0a0424118bed: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:34.044117 29862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:34.039860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55235 (local address 127.7.118.62:35927)
0929 18:02:34.040408 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:34.040424 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:34.040468 (+ 44us) server_negotiation.cc:407] Connection header received
0929 18:02:34.041200 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:34.041219 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:34.041545 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:34.041963 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:34.042807 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:34.042813 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:34.042836 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:34.042847 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:34.042901 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:34.042922 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:34.043049 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:34.043151 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 18:02:34.043328 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":153,"threads_started":1}
I20260929 18:02:34.045733 29859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eed152a1c034e8d877e0a0424118bed: No bootstrap required, opened a new log
I20260929 18:02:34.047524 7640 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:02:34.050827 29859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2eed152a1c034e8d877e0a0424118bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eed152a1c034e8d877e0a0424118bed" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35927 } }
I20260929 18:02:34.051234 29859 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2eed152a1c034e8d877e0a0424118bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:02:34.051311 29859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2eed152a1c034e8d877e0a0424118bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eed152a1c034e8d877e0a0424118bed, State: Initialized, Role: FOLLOWER
I20260929 18:02:34.051656 29862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:34.049683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34979 (local address 127.7.118.62:35927)
0929 18:02:34.049839 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:34.049844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:34.050076 (+ 232us) server_negotiation.cc:407] Connection header received
0929 18:02:34.050197 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:34.050199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:34.050238 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:34.050389 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:34.051183 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:34.051187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:34.051189 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:02:34.051200 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:34.051252 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:02:34.051270 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:34.051372 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:34.051472 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 18:02:34.051517 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 18:02:34.051996 29859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2eed152a1c034e8d877e0a0424118bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eed152a1c034e8d877e0a0424118bed" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35927 } }
I20260929 18:02:34.052201 29859 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2eed152a1c034e8d877e0a0424118bed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:02:34.052302 29859 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2eed152a1c034e8d877e0a0424118bed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:02:34.052418 29859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2eed152a1c034e8d877e0a0424118bed [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:34.053601 29859 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2eed152a1c034e8d877e0a0424118bed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eed152a1c034e8d877e0a0424118bed" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35927 } }
I20260929 18:02:34.054044 29859 leader_election.cc:302] T 00000000000000000000000000000000 P 2eed152a1c034e8d877e0a0424118bed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2eed152a1c034e8d877e0a0424118bed; no voters:
I20260929 18:02:34.054468 29859 leader_election.cc:288] T 00000000000000000000000000000000 P 2eed152a1c034e8d877e0a0424118bed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:02:34.054642 29869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2eed152a1c034e8d877e0a0424118bed [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:34.054877 29869 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2eed152a1c034e8d877e0a0424118bed [term 1 LEADER]: Becoming Leader. State: Replica: 2eed152a1c034e8d877e0a0424118bed, State: Running, Role: LEADER
I20260929 18:02:34.055254 29869 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2eed152a1c034e8d877e0a0424118bed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eed152a1c034e8d877e0a0424118bed" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35927 } }
I20260929 18:02:34.055990 29859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2eed152a1c034e8d877e0a0424118bed [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:02:34.057245 29871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eed152a1c034e8d877e0a0424118bed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2eed152a1c034e8d877e0a0424118bed. Latest consensus state: current_term: 1 leader_uuid: "2eed152a1c034e8d877e0a0424118bed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eed152a1c034e8d877e0a0424118bed" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35927 } } }
I20260929 18:02:34.057215 29870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eed152a1c034e8d877e0a0424118bed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2eed152a1c034e8d877e0a0424118bed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eed152a1c034e8d877e0a0424118bed" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 35927 } } }
I20260929 18:02:34.057379 29871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eed152a1c034e8d877e0a0424118bed [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:34.057385 29870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eed152a1c034e8d877e0a0424118bed [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:34.057777 29878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:02:34.061635 29878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:02:34.067968 29878 catalog_manager.cc:1347] Generated new cluster ID: 7105d1b30504449c946bdd33eee98a8a
I20260929 18:02:34.068077 29878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:02:34.090368 29878 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:02:34.090768 29878 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:02:34.101120 29878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2eed152a1c034e8d877e0a0424118bed: Generated new TSK 0
I20260929 18:02:34.101953 29878 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:02:34.109202 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 29791
2026-09-29T18:02:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1465 ms)
[----------] 18 tests from AuthTokenIssuingTest (26970 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:02:34 dist-test-slave-r7ph krb5kdc[29890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:02:34 dist-test-slave-r7ph krb5kdc[29890](info): set up 2 sockets
Sep 29 18:02:34 dist-test-slave-r7ph krb5kdc[29890](info): commencing operation
krb5kdc: starting...
W20260929 18:02:35.182714 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 18:02:35 dist-test-slave-r7ph krb5kdc[29890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:02:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:02:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 18:02:35.315799 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:45077
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:45077,127.7.118.61:41807,127.7.118.60:37041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:35.435331 29903 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:02:35.435721 29903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:35.435789 29903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:35.435868 29903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:35.440109 29903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:35.440212 29903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:35.440250 29903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:35.440310 29903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:35.440346 29903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:35.446135 29903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:45077,127.7.118.61:41807,127.7.118.60:37041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:45077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:35.447391 29903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:35.448488 29903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:35.457858 29909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:35.457844 29908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:35.457844 29911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:35.457901 29903 server_base.cc:1034] running on GCE node
I20260929 18:02:35.458817 29903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:35.459968 29903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:35.461159 29903 hybrid_clock.cc:648] HybridClock initialized: now 1790704955461143 us; error 33 us; skew 500 ppm
Sep 29 18:02:35 dist-test-slave-r7ph krb5kdc[29890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704955, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:35.471257 29903 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 18:02:35.472337 29903 webserver.cc:460] Webserver started at http://127.7.118.62:42129/ using document root <none> and password file <none>
I20260929 18:02:35.472954 29903 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:35.473052 29903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:35.473295 29903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:35.475515 29903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "e53243c62e4f464a860018bd8d69bb93"
format_stamp: "Formatted at 2026-09-29 18:02:35 on dist-test-slave-r7ph"
server_key: "37a11d35429796f70ba68e02e0b0c21c"
server_key_iv: "18823ad875511b6c3d18f370082ddd52"
server_key_version: "encryptionkey@0"
I20260929 18:02:35.476120 29903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "e53243c62e4f464a860018bd8d69bb93"
format_stamp: "Formatted at 2026-09-29 18:02:35 on dist-test-slave-r7ph"
server_key: "37a11d35429796f70ba68e02e0b0c21c"
server_key_iv: "18823ad875511b6c3d18f370082ddd52"
server_key_version: "encryptionkey@0"
I20260929 18:02:35.480090 29903 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 18:02:35.482746 29918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:35.484117 29903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:02:35.484259 29903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "e53243c62e4f464a860018bd8d69bb93"
format_stamp: "Formatted at 2026-09-29 18:02:35 on dist-test-slave-r7ph"
server_key: "37a11d35429796f70ba68e02e0b0c21c"
server_key_iv: "18823ad875511b6c3d18f370082ddd52"
server_key_version: "encryptionkey@0"
I20260929 18:02:35.484417 29903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:35.498670 29903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:35.502810 29903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:35.503026 29903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:35.510661 29903 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:45077
I20260929 18:02:35.510977 29970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:45077 every 8 connection(s)
I20260929 18:02:35.511770 29903 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:02:35.513192 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 29903
I20260929 18:02:35.513345 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:02:35.513660 7640 external_mini_cluster.cc:1428] Setting key 1d8b371f68bdbcdd218ca428ca9ae836
I20260929 18:02:35.515596 29971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:35.519709 29971 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45077 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.118.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.61@KRBTEST.COM" created.
Sep 29 18:02:35 dist-test-slave-r7ph krb5kdc[29890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704955, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:02:35.537665 29974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.522283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45077 (local address 127.0.0.1:55396)
0929 18:02:35.522956 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:35.523013 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:35.524143 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:35.524911 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:35.524920 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:35.525468 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:35.525888 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:35.525905 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:35.527784 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:35.527787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:35.528686 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:35.528696 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:35.529757 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:35.529763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:35.529878 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:35.530577 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:02:35.530595 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:35.532113 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:35.534532 (+ 2419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:35.534542 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:35.534557 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:35.534802 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:35.535196 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:35.535198 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:35.535200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:35.535318 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:35.536034 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:35.536039 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:35.536169 (+ 130us) client_negotiation.cc:769] Sending connection context
0929 18:02:35.536321 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 18:02:35.536798 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":120,"thread_start_us":168,"threads_started":1}
I20260929 18:02:35.537668 29975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.522286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55396 (local address 127.7.118.62:45077)
0929 18:02:35.522943 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:35.522963 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:35.523131 (+ 168us) server_negotiation.cc:407] Connection header received
0929 18:02:35.524488 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:35.524493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:35.524687 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:35.524847 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:35.526072 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:35.527639 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:35.528809 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:35.529533 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:35.529774 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:35.532296 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:35.532304 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:35.532321 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:35.532351 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:35.534399 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:35.534907 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:35.534912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:35.534913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:35.535059 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:35.535414 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:35.535416 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:35.535418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:35.535921 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:35.536034 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:35.536402 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 18:02:35.536662 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":418,"spinlock_wait_cycles":62848,"thread_start_us":172,"threads_started":1}
I20260929 18:02:35.540454 29971 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.61" port: 41807 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.61.keytab.
Entry for principal kudu/127.7.118.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.61.keytab.
I20260929 18:02:35.541523 29974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.541149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:41807 (local address 127.0.0.1:42934)
0929 18:02:35.541287 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:35.541396 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.61:41807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 18:02:35.541883 29922 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.118.61:41807: connect: Connection refused (error 111)
I20260929 18:02:35.542127 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.61:41807
--webserver_interface=127.7.118.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.61.keytab
--principal=kudu/127.7.118.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:45077,127.7.118.61:41807,127.7.118.60:37041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:02:35.543076 29974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.542773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:41807 (local address 127.0.0.1:42936)
0929 18:02:35.542961 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:35.543013 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.61:41807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260929 18:02:35.543325 29971 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.61:41807: Network error: Client connection negotiation failed: client connection to 127.7.118.61:41807: connect: Connection refused (error 111)
I20260929 18:02:35.592648 29971 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.61" port: 41807 } attempt: 1
I20260929 18:02:35.593601 29974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.593240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:41807 (local address 127.0.0.1:42940)
0929 18:02:35.593423 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:35.593504 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.61:41807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 18:02:35.594440 29974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.594180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:41807 (local address 127.0.0.1:42952)
0929 18:02:35.594314 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:35.594371 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.61:41807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 18:02:35.594623 29971 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.61:41807: Network error: Client connection negotiation failed: client connection to 127.7.118.61:41807: connect: Connection refused (error 111)
I20260929 18:02:35.662832 29971 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.61" port: 41807 } attempt: 2
I20260929 18:02:35.663834 29974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.663483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:41807 (local address 127.0.0.1:42966)
0929 18:02:35.663658 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:35.663732 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.61:41807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 18:02:35.663738 29978 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:02:35.664186 29978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:35.664268 29978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:35.664351 29978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 18:02:35.664685 29974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.664450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:41807 (local address 127.0.0.1:42980)
0929 18:02:35.664562 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:35.664616 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.61:41807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 18:02:35.664871 29971 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.61:41807: Network error: Client connection negotiation failed: client connection to 127.7.118.61:41807: connect: Connection refused (error 111)
W20260929 18:02:35.669111 29978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:35.669185 29978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:35.669206 29978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:35.669224 29978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:35.669239 29978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:35.675066 29978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:45077,127.7.118.61:41807,127.7.118.60:37041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.61.keytab
--principal=kudu/127.7.118.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.61:41807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.118.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:35.676215 29978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:35.677301 29978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:35.686513 29983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:35.686535 29984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:35.686602 29978 server_base.cc:1034] running on GCE node
W20260929 18:02:35.686559 29986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:35.687492 29978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:35.688819 29978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:35.690006 29978 hybrid_clock.cc:648] HybridClock initialized: now 1790704955689975 us; error 58 us; skew 500 ppm
Sep 29 18:02:35 dist-test-slave-r7ph krb5kdc[29890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704955, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:35.693019 29978 init.cc:377] Logged in from keytab as kudu/127.7.118.61@KRBTEST.COM (short username kudu)
I20260929 18:02:35.694027 29978 webserver.cc:460] Webserver started at http://127.7.118.61:40749/ using document root <none> and password file <none>
I20260929 18:02:35.694687 29978 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:35.694756 29978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:35.694977 29978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:35.697155 29978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-1/data/instance:
uuid: "9e6bd73e3f3444a28a6391d9246f5770"
format_stamp: "Formatted at 2026-09-29 18:02:35 on dist-test-slave-r7ph"
server_key: "3adbe7fa3ef5f5759dd0b61f1f47e0d6"
server_key_iv: "c383e315796ce5be2e313f73898fb877"
server_key_version: "encryptionkey@0"
I20260929 18:02:35.697770 29978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-1/wal/instance:
uuid: "9e6bd73e3f3444a28a6391d9246f5770"
format_stamp: "Formatted at 2026-09-29 18:02:35 on dist-test-slave-r7ph"
server_key: "3adbe7fa3ef5f5759dd0b61f1f47e0d6"
server_key_iv: "c383e315796ce5be2e313f73898fb877"
server_key_version: "encryptionkey@0"
I20260929 18:02:35.702337 29978 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 18:02:35.705456 29993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:35.707131 29978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:02:35.707302 29978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-1/wal
uuid: "9e6bd73e3f3444a28a6391d9246f5770"
format_stamp: "Formatted at 2026-09-29 18:02:35 on dist-test-slave-r7ph"
server_key: "3adbe7fa3ef5f5759dd0b61f1f47e0d6"
server_key_iv: "c383e315796ce5be2e313f73898fb877"
server_key_version: "encryptionkey@0"
I20260929 18:02:35.707446 29978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:35.728315 29978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:35.732069 29978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:35.732261 29978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:35.739632 29978 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.61:41807
I20260929 18:02:35.739952 30045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.61:41807 every 8 connection(s)
I20260929 18:02:35.740667 29978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-1/data/info.pb
I20260929 18:02:35.743665 30046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:35.747679 30046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45077 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:02:35.750017 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 29978
I20260929 18:02:35.750169 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-1/wal/instance
I20260929 18:02:35.750487 7640 external_mini_cluster.cc:1428] Setting key 10f1cdd014dfdf5fb7fa9c35356dcafc
I20260929 18:02:35.756101 29971 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.61" port: 41807 } attempt: 3
WARNING: no policy specified for kudu/127.7.118.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.60@KRBTEST.COM" created.
Sep 29 18:02:35 dist-test-slave-r7ph krb5kdc[29890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704955, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.61@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
Sep 29 18:02:35 dist-test-slave-r7ph krb5kdc[29890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704955, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for kudu/127.7.118.61@KRBTEST.COM
I20260929 18:02:35.769142 29975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.749944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55404 (local address 127.7.118.62:45077)
0929 18:02:35.750136 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:35.750142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:35.751287 (+ 1145us) server_negotiation.cc:407] Connection header received
0929 18:02:35.752661 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:35.752667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:35.752744 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:35.752856 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:35.754591 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:35.755119 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:35.757092 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:35.757519 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:35.757605 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:35.764092 (+ 6487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:35.764099 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:35.764106 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:35.764149 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:35.766672 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:35.767363 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:35.767366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:35.767368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:35.767451 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:35.767835 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:35.767840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:35.767844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:35.768085 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:35.768187 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:35.768776 (+ 589us) server_negotiation.cc:299] Negotiation successful
0929 18:02:35.768952 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 18:02:35.769701 29974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.756700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:41807 (local address 127.0.0.1:42984)
0929 18:02:35.756907 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:35.756923 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:35.757022 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:35.757740 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:35.757743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:35.757962 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:35.758055 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:35.758061 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:35.759323 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:35.759327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:35.760048 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:35.760060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:35.760706 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:35.760710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:35.760821 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:35.761328 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:02:35.761340 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:35.764122 (+ 2782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:35.766840 (+ 2718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:35.766843 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:35.766846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:35.767122 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:35.767578 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:35.767581 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:35.767585 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:35.767680 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:35.768613 (+ 933us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:35.768615 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:35.768767 (+ 152us) client_negotiation.cc:769] Sending connection context
0929 18:02:35.768858 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 18:02:35.769543 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":538}
I20260929 18:02:35.770458 30050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.756810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42984 (local address 127.7.118.61:41807)
0929 18:02:35.757284 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:35.757294 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:35.757316 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:02:35.757353 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:35.757358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:35.757543 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:35.757765 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:35.758175 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:35.759191 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:35.760183 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:35.760574 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:35.760699 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:35.764284 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:35.764290 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:35.764304 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:35.764332 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:35.766703 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:35.767257 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:35.767270 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:35.767273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:35.767463 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:35.767788 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:35.767791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:35.767793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:35.768457 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:35.769031 (+ 574us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:35.769154 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 18:02:35.769357 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"spinlock_wait_cycles":4352,"thread_start_us":193,"threads_started":1}
I20260929 18:02:35.770466 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.750455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45077 (local address 127.0.0.1:55404)
0929 18:02:35.751077 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:35.751147 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:35.752408 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:35.753039 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:35.753046 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:35.753586 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:35.754402 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:35.754437 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:35.755265 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:35.755272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:35.756910 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:35.756926 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:35.757822 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:35.757826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:35.758064 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:35.758851 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:02:35.758884 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:35.763876 (+ 4992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:35.766812 (+ 2936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:35.766827 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:35.766844 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:35.767235 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:35.767559 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:35.767566 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:35.767570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:35.767744 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:35.768226 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:35.768236 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:35.768466 (+ 230us) client_negotiation.cc:769] Sending connection context
0929 18:02:35.768696 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 18:02:35.769148 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":162,"threads_started":1}
I20260929 18:02:35.771996 30046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.61" port: 41807 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:02:35.772617 29971 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:02:35.773414 29974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.773058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:37041 (local address 127.0.0.1:51744)
0929 18:02:35.773264 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:35.773328 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260929 18:02:35.774222 29974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.773923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:37041 (local address 127.0.0.1:51756)
0929 18:02:35.774106 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:35.774159 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260929 18:02:35.774428 29971 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.60:37041: Network error: Client connection negotiation failed: client connection to 127.7.118.60:37041: connect: Connection refused (error 111)
Sep 29 18:02:35 dist-test-slave-r7ph krb5kdc[29890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704955, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.61@KRBTEST.COM for kudu/127.7.118.61@KRBTEST.COM
I20260929 18:02:35.781464 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.772599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:41807 (local address 127.0.0.1:42994)
0929 18:02:35.772847 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:35.772865 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:35.772968 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:35.773249 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:35.773252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:35.773479 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:35.773660 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:35.773671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:35.774753 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:35.774756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:35.775342 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:35.775350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:35.775883 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:35.775888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:35.776022 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:35.776548 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:02:35.776561 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:35.778041 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:35.779964 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:35.779968 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:35.779971 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:35.780170 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:35.780460 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:35.780463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:35.780465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:35.780541 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:35.780918 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:35.780921 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:35.781003 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 18:02:35.781120 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 18:02:35.781272 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260929 18:02:35.781464 30050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.772785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42994 (local address 127.7.118.61:41807)
0929 18:02:35.772941 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:35.772952 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:35.772969 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:02:35.773076 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:35.773079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:35.773133 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:35.773211 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:35.773800 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:35.774650 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:35.775472 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:35.775710 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:35.775805 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:35.778188 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:35.778191 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:35.778193 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:35.778214 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:35.779826 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:35.780286 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:35.780289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:35.780291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:35.780366 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:35.780627 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:35.780632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:35.780633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:35.780819 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:35.780932 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:35.781162 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 18:02:35.781332 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":53}
I20260929 18:02:35.782650 30046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:02:35.783448 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.783183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:37041 (local address 127.0.0.1:51772)
0929 18:02:35.783291 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:35.783382 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 18:02:35.784054 29994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.118.60:37041: connect: Connection refused (error 111)
I20260929 18:02:35.784937 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.784723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:37041 (local address 127.0.0.1:51778)
0929 18:02:35.784832 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:35.784884 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 18:02:35.785120 30046 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.60:37041: Network error: Client connection negotiation failed: client connection to 127.7.118.60:37041: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.60.keytab.
Entry for principal kudu/127.7.118.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.60.keytab.
I20260929 18:02:35.786310 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.60:37041
--webserver_interface=127.7.118.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.60.keytab
--principal=kudu/127.7.118.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:45077,127.7.118.61:41807,127.7.118.60:37041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:02:35.805614 29971 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 } attempt: 1
I20260929 18:02:35.806625 29974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.806229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:37041 (local address 127.0.0.1:51786)
0929 18:02:35.806462 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:35.806526 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260929 18:02:35.807394 29974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.807161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:37041 (local address 127.0.0.1:51798)
0929 18:02:35.807273 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:35.807330 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 18:02:35.807528 29971 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.60:37041: Network error: Client connection negotiation failed: client connection to 127.7.118.60:37041: connect: Connection refused (error 111)
I20260929 18:02:35.834446 30046 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 } attempt: 1
I20260929 18:02:35.835413 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.835096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:37041 (local address 127.0.0.1:51812)
0929 18:02:35.835266 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:35.835332 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 18:02:35.836259 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.835968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:37041 (local address 127.0.0.1:51826)
0929 18:02:35.836086 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:35.836168 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 18:02:35.836452 30046 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.60:37041: Network error: Client connection negotiation failed: client connection to 127.7.118.60:37041: connect: Connection refused (error 111)
I20260929 18:02:35.882699 29971 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 } attempt: 2
I20260929 18:02:35.883556 29974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.883267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:37041 (local address 127.0.0.1:51832)
0929 18:02:35.883417 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:35.883479 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 18:02:35.884274 29974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.884058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:37041 (local address 127.0.0.1:51840)
0929 18:02:35.884171 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:35.884223 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 18:02:35.884392 29971 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.60:37041: Network error: Client connection negotiation failed: client connection to 127.7.118.60:37041: connect: Connection refused (error 111)
W20260929 18:02:35.902510 30053 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:02:35.902875 30053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:35.902941 30053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:35.903034 30053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 18:02:35.904642 30046 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 } attempt: 2
I20260929 18:02:35.905468 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.905184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:37041 (local address 127.0.0.1:51856)
0929 18:02:35.905326 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:35.905394 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 18:02:35.906224 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.905993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:37041 (local address 127.0.0.1:51860)
0929 18:02:35.906132 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:35.906174 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 18:02:35.906436 30046 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.60:37041: Network error: Client connection negotiation failed: client connection to 127.7.118.60:37041: connect: Connection refused (error 111)
W20260929 18:02:35.907564 30053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:35.907621 30053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:35.907665 30053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:35.907703 30053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:35.907763 30053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:35.913379 30053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:45077,127.7.118.61:41807,127.7.118.60:37041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.60.keytab
--principal=kudu/127.7.118.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.60:37041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.118.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:35.914595 30053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:35.915648 30053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:35.924484 30058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:35.924526 30053 server_base.cc:1034] running on GCE node
W20260929 18:02:35.924464 30061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:35.924485 30059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:35.925348 30053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:35.926626 30053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:35.927800 30053 hybrid_clock.cc:648] HybridClock initialized: now 1790704955927781 us; error 34 us; skew 500 ppm
Sep 29 18:02:35 dist-test-slave-r7ph krb5kdc[29890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704955, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:35.930972 30053 init.cc:377] Logged in from keytab as kudu/127.7.118.60@KRBTEST.COM (short username kudu)
I20260929 18:02:35.931883 30053 webserver.cc:460] Webserver started at http://127.7.118.60:42619/ using document root <none> and password file <none>
I20260929 18:02:35.932488 30053 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:35.932605 30053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:35.932829 30053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:35.935160 30053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-2/data/instance:
uuid: "f7ebf2c2e321405d92151264bf066d21"
format_stamp: "Formatted at 2026-09-29 18:02:35 on dist-test-slave-r7ph"
server_key: "e621af11f3feeb53d0b33da90507aa46"
server_key_iv: "29777d9c5e820033c027c4bb78fde20f"
server_key_version: "encryptionkey@0"
I20260929 18:02:35.935770 30053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-2/wal/instance:
uuid: "f7ebf2c2e321405d92151264bf066d21"
format_stamp: "Formatted at 2026-09-29 18:02:35 on dist-test-slave-r7ph"
server_key: "e621af11f3feeb53d0b33da90507aa46"
server_key_iv: "29777d9c5e820033c027c4bb78fde20f"
server_key_version: "encryptionkey@0"
I20260929 18:02:35.939791 30053 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:02:35.942571 30068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:35.943984 30053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:02:35.944177 30053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-2/wal
uuid: "f7ebf2c2e321405d92151264bf066d21"
format_stamp: "Formatted at 2026-09-29 18:02:35 on dist-test-slave-r7ph"
server_key: "e621af11f3feeb53d0b33da90507aa46"
server_key_iv: "29777d9c5e820033c027c4bb78fde20f"
server_key_version: "encryptionkey@0"
I20260929 18:02:35.944316 30053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:35.968557 30053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:35.972249 30053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:35.972496 30053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:35.980654 30053 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.60:37041
I20260929 18:02:35.981042 30120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.60:37041 every 8 connection(s)
I20260929 18:02:35.981792 30053 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-2/data/info.pb
I20260929 18:02:35.983366 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 30053
I20260929 18:02:35.983570 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704657566865-7640-0/minicluster-data/master-2/wal/instance
I20260929 18:02:35.983585 29971 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 } attempt: 3
I20260929 18:02:35.983908 7640 external_mini_cluster.cc:1428] Setting key cc0b853bd9d4c179fa9917832f2d806c
I20260929 18:02:35.985589 30121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:35.990543 30121 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45077 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:02:35 dist-test-slave-r7ph krb5kdc[29890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704955, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for kudu/127.7.118.60@KRBTEST.COM
Sep 29 18:02:35 dist-test-slave-r7ph krb5kdc[29890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.60@KRBTEST.COM
I20260929 18:02:35.997598 30046 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 } attempt: 3
I20260929 18:02:35.999806 29974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.984170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:37041 (local address 127.0.0.1:51872)
0929 18:02:35.984418 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:35.984438 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:35.984552 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:35.987610 (+ 3058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:35.987630 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:35.987847 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:35.987974 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:35.987985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:35.990124 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:35.990128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:35.990969 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:35.990988 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:35.991922 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:35.991926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:35.992015 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:35.992738 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:02:35.992759 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:35.994490 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:35.997229 (+ 2739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:35.997232 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:35.997234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:35.997491 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:35.997902 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:35.997906 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:35.997908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:35.998017 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:35.999287 (+ 1270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:35.999290 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:35.999390 (+ 100us) client_negotiation.cc:769] Sending connection context
0929 18:02:35.999489 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 18:02:35.999630 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
Sep 29 18:02:35 dist-test-slave-r7ph krb5kdc[29890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704955, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.60@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:02:36.001797 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.985040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51872 (local address 127.7.118.60:37041)
0929 18:02:35.985523 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:35.985544 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:35.985592 (+ 48us) server_negotiation.cc:407] Connection header received
0929 18:02:35.986467 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:35.986495 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:35.986992 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:35.987414 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:35.988156 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:35.989950 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:35.991134 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:35.991752 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:35.992286 (+ 534us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:35.994706 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:35.994714 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:35.994729 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:35.994765 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:35.997112 (+ 2347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:35.997621 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:35.997629 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:35.997635 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:35.997786 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:35.998129 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:35.998131 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:35.998133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:35.999101 (+ 968us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:35.999236 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:36.000195 (+ 959us) server_negotiation.cc:299] Negotiation successful
0929 18:02:36.000637 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":474,"server-negotiator.queue_time_us":319,"spinlock_wait_cycles":17664,"thread_start_us":150,"threads_started":1}
I20260929 18:02:36.001797 30125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.985106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51888 (local address 127.7.118.60:37041)
0929 18:02:35.985580 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:35.985592 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:35.985602 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:02:35.986467 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:35.986495 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:35.986992 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:35.987414 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:35.988351 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:35.989950 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:35.991207 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:35.991745 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:35.991943 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:35.995554 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:35.995557 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:35.995559 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:35.995581 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:35.997448 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:35.997934 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:35.997937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:35.997938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:35.998028 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:35.998376 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:35.998382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:35.998387 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:35.999101 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:35.999997 (+ 896us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:36.000195 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 18:02:36.000687 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":730,"server-negotiator.queue_time_us":425,"spinlock_wait_cycles":93824,"thread_start_us":212,"threads_started":1}
Sep 29 18:02:36 dist-test-slave-r7ph krb5kdc[29890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704955, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.61@KRBTEST.COM for kudu/127.7.118.60@KRBTEST.COM
I20260929 18:02:36.006529 30126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.992778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:45077 (local address 127.0.0.1:55414)
0929 18:02:35.993293 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:35.993358 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:35.993548 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:35.993947 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:35.993953 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:35.994504 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:35.994676 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:35.994686 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:35.995751 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:35.995754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:35.996807 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:35.996823 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:35.997495 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:35.997499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:35.998026 (+ 527us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:35.998724 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:02:35.998738 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:36.001503 (+ 2765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:36.004124 (+ 2621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:36.004135 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:36.004141 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:36.004612 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:36.004998 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:36.005004 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:36.005006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:36.005100 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:36.005727 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:36.005734 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:36.005868 (+ 134us) client_negotiation.cc:769] Sending connection context
0929 18:02:36.006015 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 18:02:36.006330 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":140,"threads_started":1}
I20260929 18:02:36.006655 29975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.992826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55414 (local address 127.7.118.62:45077)
0929 18:02:35.993015 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:35.993020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:35.993431 (+ 411us) server_negotiation.cc:407] Connection header received
0929 18:02:35.993690 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:35.993693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:35.993757 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:35.993861 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:35.994837 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:35.995644 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:35.996958 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:35.997349 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:35.997494 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:36.001676 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:36.001679 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:36.001682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:36.001714 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:36.003987 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:36.004748 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:36.004755 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:36.004760 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:36.004856 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:36.005233 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:36.005236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:36.005238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:36.005573 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:36.005670 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:36.006357 (+ 687us) server_negotiation.cc:299] Negotiation successful
0929 18:02:36.006477 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 18:02:36.007493 29971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93: Bootstrap starting.
I20260929 18:02:36.008759 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.998105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:37041 (local address 127.0.0.1:51898)
0929 18:02:35.998452 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:35.998469 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:35.998556 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:35.999042 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:35.999047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:35.999299 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:35.999436 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:35.999445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:36.000575 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:36.000579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:36.001436 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:36.001450 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:36.002112 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:36.002117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:36.002223 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:36.002680 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:02:36.002698 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:36.004300 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:36.007061 (+ 2761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:36.007077 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:36.007079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:36.007353 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:36.007695 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:36.007699 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:36.007701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:36.007795 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:36.008198 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:36.008202 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:36.008314 (+ 112us) client_negotiation.cc:769] Sending connection context
0929 18:02:36.008429 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 18:02:36.008597 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257}
I20260929 18:02:36.008771 30127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:35.998426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51898 (local address 127.7.118.60:37041)
0929 18:02:35.998767 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:35.998775 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:35.998790 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:02:35.998828 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:35.998832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:35.998873 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:35.999560 (+ 687us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:35.999654 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:36.000442 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:36.001594 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:36.001941 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:36.002477 (+ 536us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:36.004459 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:36.004463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:36.004465 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:36.004492 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:36.006939 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:36.007486 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:36.007489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:36.007490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:36.007588 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:36.007901 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:36.007904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:36.007905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:36.008117 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:36.008206 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:36.008491 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 18:02:36.008604 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":151,"threads_started":1}
I20260929 18:02:36.010658 30121 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.61" port: 41807 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:02:36 dist-test-slave-r7ph krb5kdc[29890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:02:36.012177 29971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:36.012665 30046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e6bd73e3f3444a28a6391d9246f5770: Bootstrap starting.
I20260929 18:02:36.013235 29971 log.cc:824] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:36.015698 29971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93: No bootstrap required, opened a new log
I20260929 18:02:36.015676 30046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e6bd73e3f3444a28a6391d9246f5770: Neither blocks nor log segments found. Creating new log.
Sep 29 18:02:36 dist-test-slave-r7ph krb5kdc[29890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704955, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.60@KRBTEST.COM for kudu/127.7.118.61@KRBTEST.COM
I20260929 18:02:36.016467 30046 log.cc:824] T 00000000000000000000000000000000 P 9e6bd73e3f3444a28a6391d9246f5770: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:36.019130 30046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e6bd73e3f3444a28a6391d9246f5770: No bootstrap required, opened a new log
I20260929 18:02:36.020629 29971 raft_consensus.cc:348] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53243c62e4f464a860018bd8d69bb93" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45077 } } peers { permanent_uuid: "9e6bd73e3f3444a28a6391d9246f5770" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 41807 } } peers { permanent_uuid: "f7ebf2c2e321405d92151264bf066d21" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 } }
I20260929 18:02:36.020920 29971 raft_consensus.cc:729] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53243c62e4f464a860018bd8d69bb93, State: Initialized, Role: FOLLOWER
I20260929 18:02:36.021123 30050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:36.011425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43002 (local address 127.7.118.61:41807)
0929 18:02:36.011558 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:36.011562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:36.011574 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:02:36.011615 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:36.011616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:36.011650 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:36.011739 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:36.012209 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:36.012950 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:36.014016 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:36.014369 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:36.014477 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:36.017002 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:36.017006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:36.017009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:36.017036 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:36.019086 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:36.019621 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:36.019625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:36.019627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:36.019713 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:36.020059 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:36.020062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:36.020064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:36.020346 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:36.020420 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:36.020814 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 18:02:36.020968 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:02:36.021279 29971 consensus_queue.cc:260] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53243c62e4f464a860018bd8d69bb93" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45077 } } peers { permanent_uuid: "9e6bd73e3f3444a28a6391d9246f5770" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 41807 } } peers { permanent_uuid: "f7ebf2c2e321405d92151264bf066d21" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 } }
I20260929 18:02:36.021672 30126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:36.011256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:41807 (local address 127.0.0.1:43002)
0929 18:02:36.011365 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:36.011380 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:36.011460 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:36.011754 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:36.011757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:36.011938 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:36.012064 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:36.012073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:36.013094 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:36.013098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:36.013868 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:36.013881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:36.014509 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:36.014512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:36.014603 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:36.015170 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:02:36.015186 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:36.016827 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:36.019226 (+ 2399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:36.019230 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:36.019232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:36.019500 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:36.019845 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:36.019848 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:36.019850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:36.019939 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:36.020498 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:36.020501 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:36.020603 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 18:02:36.021316 (+ 713us) client_negotiation.cc:240] Negotiation successful
0929 18:02:36.021478 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 18:02:36.022480 30121 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:02:36.022550 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:36.005195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55420 (local address 127.7.118.62:45077)
0929 18:02:36.005554 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:36.005585 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:36.005600 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:02:36.005638 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:36.005642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:36.005714 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:36.007095 (+ 1381us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:36.007213 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:36.008045 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:36.009151 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:36.009494 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:36.009650 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:36.017101 (+ 7451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:36.017105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:36.017108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:36.017128 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:36.019331 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:36.019933 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:36.019936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:36.019938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:36.020048 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:36.020404 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:36.020407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:36.020409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:36.020632 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:36.022097 (+ 1465us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:36.022183 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 18:02:36.022355 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":78,"threads_started":1}
I20260929 18:02:36.022897 29971 sys_catalog.cc:564] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:02:36.023792 30046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e6bd73e3f3444a28a6391d9246f5770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53243c62e4f464a860018bd8d69bb93" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45077 } } peers { permanent_uuid: "9e6bd73e3f3444a28a6391d9246f5770" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 41807 } } peers { permanent_uuid: "f7ebf2c2e321405d92151264bf066d21" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 } }
I20260929 18:02:36.024050 30046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e6bd73e3f3444a28a6391d9246f5770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e6bd73e3f3444a28a6391d9246f5770, State: Initialized, Role: FOLLOWER
I20260929 18:02:36.024397 30046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e6bd73e3f3444a28a6391d9246f5770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53243c62e4f464a860018bd8d69bb93" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45077 } } peers { permanent_uuid: "9e6bd73e3f3444a28a6391d9246f5770" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 41807 } } peers { permanent_uuid: "f7ebf2c2e321405d92151264bf066d21" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 } }
I20260929 18:02:36.026142 30046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e6bd73e3f3444a28a6391d9246f5770 [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 18:02:36 dist-test-slave-r7ph krb5kdc[29890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704955, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.60@KRBTEST.COM for kudu/127.7.118.60@KRBTEST.COM
I20260929 18:02:36.028455 30141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e6bd73e3f3444a28a6391d9246f5770 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53243c62e4f464a860018bd8d69bb93" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45077 } } peers { permanent_uuid: "9e6bd73e3f3444a28a6391d9246f5770" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 41807 } } peers { permanent_uuid: "f7ebf2c2e321405d92151264bf066d21" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 } } }
I20260929 18:02:36.028853 30141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e6bd73e3f3444a28a6391d9246f5770 [sys.catalog]: This master's current role is: FOLLOWER
Sep 29 18:02:36 dist-test-slave-r7ph krb5kdc[29890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.61@KRBTEST.COM
W20260929 18:02:36.032760 30155 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e6bd73e3f3444a28a6391d9246f5770: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:02:36.032866 30155 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:02:36.033152 30131 sys_catalog.cc:455] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53243c62e4f464a860018bd8d69bb93" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45077 } } peers { permanent_uuid: "9e6bd73e3f3444a28a6391d9246f5770" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 41807 } } peers { permanent_uuid: "f7ebf2c2e321405d92151264bf066d21" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 } } }
I20260929 18:02:36.029302 29903 master_runner.cc:186] Error getting master registration for 127.7.118.61:41807: OK, instance_id { permanent_uuid: "9e6bd73e3f3444a28a6391d9246f5770" instance_seqno: 1790704955737526 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:02:36.033383 29903 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:02:36.033447 30131 sys_catalog.cc:458] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:02:36.034324 30127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:36.023488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51904 (local address 127.7.118.60:37041)
0929 18:02:36.023585 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:36.023589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:36.023598 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:02:36.023629 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:36.023631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:36.023676 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:36.023753 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:36.024303 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:36.025107 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:36.025927 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:36.026278 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:36.026354 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:36.028845 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:36.028850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:36.028853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:36.028875 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:36.031971 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:36.032595 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:36.032599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:36.032601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:36.032678 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:36.033086 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:36.033089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:36.033091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:36.033590 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:36.033669 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:36.033967 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 18:02:36.034115 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 18:02:36.035020 30126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:36.023002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:37041 (local address 127.0.0.1:51904)
0929 18:02:36.023319 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:36.023334 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:36.023409 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:36.023884 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:36.023888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:36.024084 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:36.024183 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:36.024194 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:36.025226 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:36.025228 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:36.025786 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:36.025793 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:36.026492 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:36.026497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:36.026568 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:36.027110 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:02:36.027125 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:36.028686 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:36.032256 (+ 3570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:36.032259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:36.032261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:36.032496 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:36.032773 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:36.032775 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:36.032777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:36.032968 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:36.033729 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:36.033731 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:36.033821 (+ 90us) client_negotiation.cc:769] Sending connection context
0929 18:02:36.034727 (+ 906us) client_negotiation.cc:240] Negotiation successful
0929 18:02:36.034864 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":108,"threads_started":1}
I20260929 18:02:36.036001 30050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:36.024671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43018 (local address 127.7.118.61:41807)
0929 18:02:36.024881 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:36.024885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:36.024897 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:02:36.024932 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:36.024934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:36.024972 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:36.025862 (+ 890us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:36.025953 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:36.026776 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:36.027990 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:36.028445 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:36.028547 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:36.031937 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:36.031941 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:36.031944 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:36.031970 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:36.033652 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:36.034168 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:36.034172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:36.034173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:36.034297 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:36.034612 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:36.034616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:36.034617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:36.034849 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:36.035010 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:36.035719 (+ 709us) server_negotiation.cc:299] Negotiation successful
0929 18:02:36.035848 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260929 18:02:36.038450 30121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ebf2c2e321405d92151264bf066d21: Bootstrap starting.
I20260929 18:02:36.037811 29978 master_runner.cc:186] Error getting master registration for 127.7.118.60:37041: OK, instance_id { permanent_uuid: "f7ebf2c2e321405d92151264bf066d21" instance_seqno: 1790704955978318 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:02:36.041002 29978 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:02:36.041447 30121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7ebf2c2e321405d92151264bf066d21: Neither blocks nor log segments found. Creating new log.
W20260929 18:02:36.042121 30144 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:02:36.042194 30144 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:02:36.042680 7640 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:02:36.042722 30121 log.cc:824] T 00000000000000000000000000000000 P f7ebf2c2e321405d92151264bf066d21: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:36.044252 29945 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e53243c62e4f464a860018bd8d69bb93"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55420
I20260929 18:02:36.044508 29945 raft_consensus.cc:482] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 18:02:36.044637 29945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:36.046139 29945 raft_consensus.cc:504] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53243c62e4f464a860018bd8d69bb93" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45077 } } peers { permanent_uuid: "9e6bd73e3f3444a28a6391d9246f5770" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 41807 } } peers { permanent_uuid: "f7ebf2c2e321405d92151264bf066d21" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 } }
I20260929 18:02:36.048036 29945 leader_election.cc:288] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93 [CANDIDATE]: Term 1 election: Requested vote from peers 9e6bd73e3f3444a28a6391d9246f5770 (127.7.118.61:41807), f7ebf2c2e321405d92151264bf066d21 (127.7.118.60:37041)
I20260929 18:02:36.048054 30020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e53243c62e4f464a860018bd8d69bb93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9e6bd73e3f3444a28a6391d9246f5770"
I20260929 18:02:36.048352 30020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e6bd73e3f3444a28a6391d9246f5770 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:36.050038 30020 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9e6bd73e3f3444a28a6391d9246f5770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e53243c62e4f464a860018bd8d69bb93 in term 1.
I20260929 18:02:36.050669 30121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ebf2c2e321405d92151264bf066d21: No bootstrap required, opened a new log
I20260929 18:02:36.050653 29922 leader_election.cc:302] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e6bd73e3f3444a28a6391d9246f5770, e53243c62e4f464a860018bd8d69bb93; no voters:
I20260929 18:02:36.051065 30131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:36.051302 30131 raft_consensus.cc:686] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93 [term 1 LEADER]: Becoming Leader. State: Replica: e53243c62e4f464a860018bd8d69bb93, State: Running, Role: LEADER
I20260929 18:02:36.051693 30131 consensus_queue.cc:237] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53243c62e4f464a860018bd8d69bb93" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45077 } } peers { permanent_uuid: "9e6bd73e3f3444a28a6391d9246f5770" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 41807 } } peers { permanent_uuid: "f7ebf2c2e321405d92151264bf066d21" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 } }
I20260929 18:02:36.053687 30131 sys_catalog.cc:455] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e53243c62e4f464a860018bd8d69bb93. Latest consensus state: current_term: 1 leader_uuid: "e53243c62e4f464a860018bd8d69bb93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53243c62e4f464a860018bd8d69bb93" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45077 } } peers { permanent_uuid: "9e6bd73e3f3444a28a6391d9246f5770" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 41807 } } peers { permanent_uuid: "f7ebf2c2e321405d92151264bf066d21" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 } } }
I20260929 18:02:36.053819 30131 sys_catalog.cc:458] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:36.056466 30166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:02:36.056277 30121 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7ebf2c2e321405d92151264bf066d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53243c62e4f464a860018bd8d69bb93" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45077 } } peers { permanent_uuid: "9e6bd73e3f3444a28a6391d9246f5770" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 41807 } } peers { permanent_uuid: "f7ebf2c2e321405d92151264bf066d21" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 } }
I20260929 18:02:36.056587 30121 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7ebf2c2e321405d92151264bf066d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ebf2c2e321405d92151264bf066d21, State: Initialized, Role: FOLLOWER
I20260929 18:02:36.057003 30121 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7ebf2c2e321405d92151264bf066d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53243c62e4f464a860018bd8d69bb93" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45077 } } peers { permanent_uuid: "9e6bd73e3f3444a28a6391d9246f5770" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 41807 } } peers { permanent_uuid: "f7ebf2c2e321405d92151264bf066d21" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 } }
I20260929 18:02:36.058496 30166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:02:36.058666 30121 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7ebf2c2e321405d92151264bf066d21 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:02:36.058733 30095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e53243c62e4f464a860018bd8d69bb93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f7ebf2c2e321405d92151264bf066d21"
I20260929 18:02:36.058995 30095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7ebf2c2e321405d92151264bf066d21 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:36.060463 30095 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f7ebf2c2e321405d92151264bf066d21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e53243c62e4f464a860018bd8d69bb93 in term 1.
I20260929 18:02:36.074133 30050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:36.050965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43028 (local address 127.7.118.61:41807)
0929 18:02:36.051107 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:36.051112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:36.061997 (+ 10885us) server_negotiation.cc:407] Connection header received
0929 18:02:36.062683 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:36.062687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:36.062749 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:36.062846 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:36.063930 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:36.064744 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:36.066187 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:36.066580 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:36.066668 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:36.069823 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:36.069828 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:36.069832 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:36.069870 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:36.071891 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:36.072480 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:36.072485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:36.072487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:36.072593 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:36.072915 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:36.072929 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:36.072931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:36.073177 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:36.073290 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:36.073781 (+ 491us) server_negotiation.cc:299] Negotiation successful
0929 18:02:36.073940 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:02:36.074506 30167 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ebf2c2e321405d92151264bf066d21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53243c62e4f464a860018bd8d69bb93" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45077 } } peers { permanent_uuid: "9e6bd73e3f3444a28a6391d9246f5770" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 41807 } } peers { permanent_uuid: "f7ebf2c2e321405d92151264bf066d21" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 } } }
I20260929 18:02:36.074733 30167 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ebf2c2e321405d92151264bf066d21 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 18:02:36.078693 30180 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7ebf2c2e321405d92151264bf066d21: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:02:36.078797 30180 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:02:36.079860 30127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:36.059475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51912 (local address 127.7.118.60:37041)
0929 18:02:36.060796 (+ 1321us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:36.060801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:36.064406 (+ 3605us) server_negotiation.cc:407] Connection header received
0929 18:02:36.064517 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:36.064520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:36.064566 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:36.064633 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:36.065543 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:36.066308 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:36.068261 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:36.068623 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:36.068696 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:36.073843 (+ 5147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:36.073848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:36.073850 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:36.073882 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:36.076752 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:36.078422 (+ 1670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:36.078426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:36.078428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:36.078527 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:36.078824 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:36.078827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:36.078829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:36.079060 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:36.079171 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:36.079570 (+ 399us) server_negotiation.cc:299] Negotiation successful
0929 18:02:36.079699 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1244}
I20260929 18:02:36.085692 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:36.058685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55432 (local address 127.7.118.62:45077)
0929 18:02:36.058830 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:36.058836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:36.063961 (+ 5125us) server_negotiation.cc:407] Connection header received
0929 18:02:36.064153 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:36.064156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:36.064209 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:36.064264 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:36.069370 (+ 5106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:36.070174 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:36.072328 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:36.072758 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:36.072842 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:36.081583 (+ 8741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:36.081588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:36.081591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:36.081627 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:36.083763 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:36.084294 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:36.084299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:36.084300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:36.084401 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:36.084694 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:36.084697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:36.084699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:36.084922 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:36.085032 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:36.085397 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 18:02:36.085528 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:02:36.091709 30095 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7ebf2c2e321405d92151264bf066d21 [term 1 FOLLOWER]: Refusing update from remote peer e53243c62e4f464a860018bd8d69bb93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:02:36.092554 30163 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ebf2c2e321405d92151264bf066d21" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:02:36.099820 30167 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ebf2c2e321405d92151264bf066d21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e53243c62e4f464a860018bd8d69bb93. Latest consensus state: current_term: 1 leader_uuid: "e53243c62e4f464a860018bd8d69bb93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53243c62e4f464a860018bd8d69bb93" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45077 } } peers { permanent_uuid: "9e6bd73e3f3444a28a6391d9246f5770" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 41807 } } peers { permanent_uuid: "f7ebf2c2e321405d92151264bf066d21" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 } } }
I20260929 18:02:36.100004 30167 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ebf2c2e321405d92151264bf066d21 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:02:36.103951 30020 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9e6bd73e3f3444a28a6391d9246f5770 [term 1 FOLLOWER]: Refusing update from remote peer e53243c62e4f464a860018bd8d69bb93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:02:36.104665 30163 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e6bd73e3f3444a28a6391d9246f5770" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 41807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:02:36.109465 30141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e6bd73e3f3444a28a6391d9246f5770 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e53243c62e4f464a860018bd8d69bb93. Latest consensus state: current_term: 1 leader_uuid: "e53243c62e4f464a860018bd8d69bb93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53243c62e4f464a860018bd8d69bb93" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45077 } } peers { permanent_uuid: "9e6bd73e3f3444a28a6391d9246f5770" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 41807 } } peers { permanent_uuid: "f7ebf2c2e321405d92151264bf066d21" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 } } }
I20260929 18:02:36.109645 30141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e6bd73e3f3444a28a6391d9246f5770 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:02:36.121353 30131 sys_catalog.cc:455] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e53243c62e4f464a860018bd8d69bb93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53243c62e4f464a860018bd8d69bb93" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45077 } } peers { permanent_uuid: "9e6bd73e3f3444a28a6391d9246f5770" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 41807 } } peers { permanent_uuid: "f7ebf2c2e321405d92151264bf066d21" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 } } }
I20260929 18:02:36.121541 30131 sys_catalog.cc:458] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:36.121754 30141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e6bd73e3f3444a28a6391d9246f5770 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e53243c62e4f464a860018bd8d69bb93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53243c62e4f464a860018bd8d69bb93" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45077 } } peers { permanent_uuid: "9e6bd73e3f3444a28a6391d9246f5770" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 41807 } } peers { permanent_uuid: "f7ebf2c2e321405d92151264bf066d21" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 } } }
I20260929 18:02:36.121865 30181 mvcc.cc:205] Tried to move back new op lower bound from 7334727500142583808 to 7334727499991769088. Current Snapshot: MvccSnapshot[applied={T|T < 7334727500142583808}]
I20260929 18:02:36.121897 30141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e6bd73e3f3444a28a6391d9246f5770 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:02:36.125322 30185 mvcc.cc:205] Tried to move back new op lower bound from 7334727500142583808 to 7334727499991769088. Current Snapshot: MvccSnapshot[applied={T|T < 7334727500142583808}]
I20260929 18:02:36.126179 30163 sys_catalog.cc:455] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e53243c62e4f464a860018bd8d69bb93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53243c62e4f464a860018bd8d69bb93" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45077 } } peers { permanent_uuid: "9e6bd73e3f3444a28a6391d9246f5770" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 41807 } } peers { permanent_uuid: "f7ebf2c2e321405d92151264bf066d21" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 } } }
I20260929 18:02:36.126335 30163 sys_catalog.cc:458] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93 [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:36.127686 30167 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ebf2c2e321405d92151264bf066d21 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e53243c62e4f464a860018bd8d69bb93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53243c62e4f464a860018bd8d69bb93" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 45077 } } peers { permanent_uuid: "9e6bd73e3f3444a28a6391d9246f5770" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 41807 } } peers { permanent_uuid: "f7ebf2c2e321405d92151264bf066d21" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 37041 } } }
I20260929 18:02:36.127825 30167 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ebf2c2e321405d92151264bf066d21 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:02:36.128671 30183 mvcc.cc:205] Tried to move back new op lower bound from 7334727500142583808 to 7334727499991769088. Current Snapshot: MvccSnapshot[applied={T|T < 7334727500142583808}]
I20260929 18:02:36.135372 30166 catalog_manager.cc:1347] Generated new cluster ID: 73dde972d3f3430f861ba119697a2431
I20260929 18:02:36.135474 30166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:02:36.173883 30166 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:02:36.175344 30166 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:02:36.196188 30166 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e53243c62e4f464a860018bd8d69bb93: Generated new TSK 0
I20260929 18:02:36.197167 30166 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:02:36.234081 30050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:02:36.232073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43034 (local address 127.7.118.61:41807)
0929 18:02:36.232206 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:36.232211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:36.232902 (+ 691us) server_negotiation.cc:407] Connection header received
0929 18:02:36.233125 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:36.233129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:36.233183 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:36.233242 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:36.233940 (+ 698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43034: BlockingRecv error: recv got EOF from 127.0.0.1:43034 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 18:02:36.236687 30128 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:02:36.234855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55444 (local address 127.7.118.62:45077)
0929 18:02:36.234972 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:36.234977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:36.235698 (+ 721us) server_negotiation.cc:407] Connection header received
0929 18:02:36.235879 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:36.235883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:36.235934 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:36.235988 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:36.236555 (+ 567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55444: BlockingRecv error: recv got EOF from 127.0.0.1:55444 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 18:02:36.238153 30127 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:02:36.235407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51924 (local address 127.7.118.60:37041)
0929 18:02:36.235520 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:36.235524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:36.237290 (+ 1766us) server_negotiation.cc:407] Connection header received
0929 18:02:36.237449 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:36.237453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:36.237501 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:36.237561 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:36.238030 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51924: BlockingRecv error: recv got EOF from 127.0.0.1:51924 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 18:02:36.242571 30050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:02:36.240947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43050 (local address 127.7.118.61:41807)
0929 18:02:36.241055 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:36.241059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:36.241550 (+ 491us) server_negotiation.cc:407] Connection header received
0929 18:02:36.241765 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:36.241769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:36.241842 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:36.241894 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:36.242443 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43050: BlockingRecv error: recv got EOF from 127.0.0.1:43050 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 18:02:36.248041 30050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:02:36.246175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43052 (local address 127.7.118.61:41807)
0929 18:02:36.246314 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:36.246319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:36.246891 (+ 572us) server_negotiation.cc:407] Connection header received
0929 18:02:36.247088 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:36.247092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:36.247147 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:36.247200 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:36.247870 (+ 670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43052: BlockingRecv error: recv got EOF from 127.0.0.1:43052 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 18:02:36.254093 30050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:02:36.252258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43060 (local address 127.7.118.61:41807)
0929 18:02:36.252378 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:36.252383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:36.252963 (+ 580us) server_negotiation.cc:407] Connection header received
0929 18:02:36.253215 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:36.253220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:36.253297 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:36.253361 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:36.253969 (+ 608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43060: BlockingRecv error: recv got EOF from 127.0.0.1:43060 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 18:02:36.264930 30050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:02:36.262825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43074 (local address 127.7.118.61:41807)
0929 18:02:36.262993 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:36.262998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:36.263678 (+ 680us) server_negotiation.cc:407] Connection header received
0929 18:02:36.263892 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:36.263896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:36.263953 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:36.264011 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:36.264805 (+ 794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43074: BlockingRecv error: recv got EOF from 127.0.0.1:43074 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260929 18:02:36.277042 30050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:02:36.275683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43088 (local address 127.7.118.61:41807)
0929 18:02:36.275834 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:36.275838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:36.276106 (+ 268us) server_negotiation.cc:407] Connection header received
0929 18:02:36.276281 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:36.276285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:36.276349 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:36.276444 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:36.276910 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43088: BlockingRecv error: recv got EOF from 127.0.0.1:43088 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 18:02:36.296545 30050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:02:36.295126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43094 (local address 127.7.118.61:41807)
0929 18:02:36.295267 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:36.295272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:36.295549 (+ 277us) server_negotiation.cc:407] Connection header received
0929 18:02:36.295734 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:36.295744 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:36.295800 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:36.295896 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:36.296422 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43094: BlockingRecv error: recv got EOF from 127.0.0.1:43094 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 18:02:36.332345 30050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:02:36.330989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43106 (local address 127.7.118.61:41807)
0929 18:02:36.331162 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:36.331166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:36.331200 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:02:36.331390 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:36.331393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:36.331460 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:36.331541 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:36.332171 (+ 630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43106: BlockingRecv error: recv got EOF from 127.0.0.1:43106 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260929 18:02:36.399544 30050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:02:36.398472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43316 (local address 127.7.118.61:41807)
0929 18:02:36.398629 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:36.398633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:36.398776 (+ 143us) server_negotiation.cc:407] Connection header received
0929 18:02:36.398903 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:36.398906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:36.398955 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:36.399039 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:36.399437 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43316: BlockingRecv error: recv got EOF from 127.0.0.1:43316 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 18:02:36.531325 30050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:02:36.529964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43324 (local address 127.7.118.61:41807)
0929 18:02:36.530164 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:36.530168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:36.530301 (+ 133us) server_negotiation.cc:407] Connection header received
0929 18:02:36.530520 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:36.530522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:36.530573 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:36.530646 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:36.531203 (+ 557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43324: BlockingRecv error: recv got EOF from 127.0.0.1:43324 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 18:02:36.790898 30050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:02:36.789704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43332 (local address 127.7.118.61:41807)
0929 18:02:36.789876 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:36.789881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:36.789914 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:02:36.790111 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:36.790114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:36.790171 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:36.790286 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:36.790778 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43332: BlockingRecv error: recv got EOF from 127.0.0.1:43332 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 18:02:37.034226 30155 catalog_manager.cc:1259] Loaded cluster ID: 73dde972d3f3430f861ba119697a2431
I20260929 18:02:37.034370 30155 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9e6bd73e3f3444a28a6391d9246f5770: loading cluster ID for follower catalog manager: success
I20260929 18:02:37.036171 30155 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9e6bd73e3f3444a28a6391d9246f5770: acquiring CA information for follower catalog manager: success
I20260929 18:02:37.037777 30155 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9e6bd73e3f3444a28a6391d9246f5770: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:02:37.080190 30180 catalog_manager.cc:1259] Loaded cluster ID: 73dde972d3f3430f861ba119697a2431
I20260929 18:02:37.080327 30180 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f7ebf2c2e321405d92151264bf066d21: loading cluster ID for follower catalog manager: success
I20260929 18:02:37.082055 30180 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f7ebf2c2e321405d92151264bf066d21: acquiring CA information for follower catalog manager: success
I20260929 18:02:37.083362 30180 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f7ebf2c2e321405d92151264bf066d21: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:02:37.310004 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:37.305228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43342 (local address 127.7.118.61:41807)
0929 18:02:37.305546 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:37.305551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:37.305703 (+ 152us) server_negotiation.cc:407] Connection header received
0929 18:02:37.305971 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:37.305974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:37.306105 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:37.306283 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:02:37.306600 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:37.307209 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:37.308239 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:37.308506 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:37.308635 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:37.309044 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:02:37.309532 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:02:37.309694 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:37.309735 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 18:02:37.309801 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":78,"threads_started":1}
I20260929 18:02:37.318457 30258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:37.313801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48328 (local address 127.7.118.60:37041)
0929 18:02:37.314081 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:37.314086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:37.314227 (+ 141us) server_negotiation.cc:407] Connection header received
0929 18:02:37.314499 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:37.314502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:37.314632 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:37.314779 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:02:37.315052 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:37.315674 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:37.316702 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:37.316963 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:37.317088 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:37.317456 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:02:37.317953 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:02:37.318119 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:37.318163 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 18:02:37.318232 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":71,"threads_started":1}
I20260929 18:02:37.327569 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:37.322119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43350 (local address 127.7.118.61:41807)
0929 18:02:37.322306 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:37.322313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:37.322799 (+ 486us) server_negotiation.cc:407] Connection header received
0929 18:02:37.323170 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:37.323174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:37.323235 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:37.323313 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:02:37.323762 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:37.324608 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:37.326143 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:37.326550 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:37.326653 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:37.326951 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:02:37.327053 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:02:37.327126 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:37.327312 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 18:02:37.327408 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:02:37.327947 30258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:37.322555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48336 (local address 127.7.118.60:37041)
0929 18:02:37.322677 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:37.322681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:37.322928 (+ 247us) server_negotiation.cc:407] Connection header received
0929 18:02:37.323225 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:37.323229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:37.323317 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:37.323427 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:02:37.324758 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:37.325398 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:37.326575 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:37.326916 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:37.327043 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:37.327429 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:02:37.327518 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:02:37.327563 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:37.327743 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 18:02:37.327816 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:02:37.328377 30265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:37.322105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43466 (local address 127.7.118.62:45077)
0929 18:02:37.322411 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:37.322415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:37.322791 (+ 376us) server_negotiation.cc:407] Connection header received
0929 18:02:37.323015 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:37.323018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:37.323182 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:37.323324 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:02:37.323606 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:37.324599 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:37.326395 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:37.326707 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:37.326827 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:37.327306 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:02:37.327718 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:02:37.327808 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:37.327949 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 18:02:37.328189 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":104,"threads_started":1}
I20260929 18:02:37.333153 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 29903
I20260929 18:02:37.339877 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 29978
I20260929 18:02:37.346134 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 30053
2026-09-29T18:02:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3239 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:02:37 dist-test-slave-r7ph krb5kdc[30269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:02:37 dist-test-slave-r7ph krb5kdc[30269](info): set up 2 sockets
Sep 29 18:02:37 dist-test-slave-r7ph krb5kdc[30269](info): commencing operation
krb5kdc: starting...
W20260929 18:02:38.444646 7640 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/krb5kdc/test-user.keytab.
Sep 29 18:02:38 dist-test-slave-r7ph krb5kdc[30269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:02:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:02:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.118.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
Entry for principal kudu/127.7.118.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab.
I20260929 18:02:38.590345 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:42283
--webserver_interface=127.7.118.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:35705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:42283,127.7.118.61:40801,127.7.118.60:42509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:02:38.715679 30282 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:02:38.716106 30282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:38.716149 30282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:38.716231 30282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:38.720639 30282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:38.720722 30282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:38.720743 30282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:38.720762 30282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:38.720777 30282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:38.726562 30282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:35705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:42283,127.7.118.61:40801,127.7.118.60:42509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.62.keytab
--principal=kudu/127.7.118.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.62:42283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.118.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:38.727658 30282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:38.728690 30282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:38.738075 30287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:38.738044 30288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:38.738389 30282 server_base.cc:1034] running on GCE node
W20260929 18:02:38.738044 30290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:38.739075 30282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:38.740415 30282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:38.741660 30282 hybrid_clock.cc:648] HybridClock initialized: now 1790704958741638 us; error 41 us; skew 500 ppm
Sep 29 18:02:38 dist-test-slave-r7ph krb5kdc[30269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704958, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:38.745177 30282 init.cc:377] Logged in from keytab as kudu/127.7.118.62@KRBTEST.COM (short username kudu)
I20260929 18:02:38.746472 30282 webserver.cc:460] Webserver started at http://127.7.118.62:42353/ using document root <none> and password file <none>
I20260929 18:02:38.747166 30282 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:38.747272 30282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:38.747539 30282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:38.749853 30282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-0/data/instance:
uuid: "10bc9605c80e41a7b6eccf67f530977c"
format_stamp: "Formatted at 2026-09-29 18:02:38 on dist-test-slave-r7ph"
server_key: "3f2caf4e3b0fc10ed1470e4d1161b1a6"
server_key_iv: "536b62bfe391757d05f51076995f169f"
server_key_version: "encryptionkey@0"
I20260929 18:02:38.750551 30282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-0/wal/instance:
uuid: "10bc9605c80e41a7b6eccf67f530977c"
format_stamp: "Formatted at 2026-09-29 18:02:38 on dist-test-slave-r7ph"
server_key: "3f2caf4e3b0fc10ed1470e4d1161b1a6"
server_key_iv: "536b62bfe391757d05f51076995f169f"
server_key_version: "encryptionkey@0"
I20260929 18:02:38.755219 30282 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 18:02:38.758359 30297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:38.759893 30282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:02:38.760089 30282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-0/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-0/wal
uuid: "10bc9605c80e41a7b6eccf67f530977c"
format_stamp: "Formatted at 2026-09-29 18:02:38 on dist-test-slave-r7ph"
server_key: "3f2caf4e3b0fc10ed1470e4d1161b1a6"
server_key_iv: "536b62bfe391757d05f51076995f169f"
server_key_version: "encryptionkey@0"
I20260929 18:02:38.760242 30282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:38.789988 30282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:38.794039 30282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:38.794308 30282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:38.802448 30282 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.62:42283
I20260929 18:02:38.802835 30349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.62:42283 every 8 connection(s)
I20260929 18:02:38.803591 30282 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-0/data/info.pb
I20260929 18:02:38.806960 30350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:38.808034 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 30282
I20260929 18:02:38.808168 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-0/wal/instance
I20260929 18:02:38.808504 7640 external_mini_cluster.cc:1428] Setting key 150685641125eb24fb6d24673b4b9b8c
I20260929 18:02:38.812336 30350 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42283 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.118.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.61@KRBTEST.COM" created.
Sep 29 18:02:38 dist-test-slave-r7ph krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704958, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:02:38.832173 30354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:38.815099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54860 (local address 127.7.118.62:42283)
0929 18:02:38.815740 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:38.815752 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:38.815871 (+ 119us) server_negotiation.cc:407] Connection header received
0929 18:02:38.817112 (+ 1241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:38.817118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:38.817341 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:38.817570 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:38.818714 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:38.820056 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:38.821926 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:38.822691 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:38.822928 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:38.825982 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:38.825991 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:38.826008 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:38.826044 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:38.828334 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:38.828957 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:38.828967 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:38.828971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:38.829112 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:38.829500 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:38.829503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:38.829504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:38.830067 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:38.830189 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:38.830879 (+ 690us) server_negotiation.cc:299] Negotiation successful
0929 18:02:38.831207 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":181,"server-negotiator.queue_time_us":511,"spinlock_wait_cycles":39552,"thread_start_us":176,"threads_started":1}
I20260929 18:02:38.832188 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:38.815099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42283 (local address 127.0.0.1:54860)
0929 18:02:38.815660 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:38.815717 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:38.816792 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:38.817635 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:38.817645 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:38.818104 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:38.818489 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:38.818508 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:38.820213 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:38.820221 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:38.821767 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:38.821781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:38.822924 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:38.822928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:38.823080 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:38.823908 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:02:38.823929 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:38.825770 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:38.828500 (+ 2730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:38.828510 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:38.828520 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:38.828817 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:38.829217 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:38.829220 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:38.829221 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:38.829382 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:38.830215 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:38.830221 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:38.830520 (+ 299us) client_negotiation.cc:769] Sending connection context
0929 18:02:38.830728 (+ 208us) client_negotiation.cc:240] Negotiation successful
0929 18:02:38.831069 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":6,"spinlock_wait_cycles":7040,"thread_start_us":182,"threads_started":1}
I20260929 18:02:38.835276 30350 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.61" port: 40801 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:02:38.836241 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:38.835909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:40801 (local address 127.0.0.1:35098)
0929 18:02:38.836077 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:38.836155 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.61:40801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 18:02:38.836608 30300 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.118.61:40801: connect: Connection refused (error 111)
I20260929 18:02:38.837736 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:38.837490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:40801 (local address 127.0.0.1:35108)
0929 18:02:38.837619 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:38.837672 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.61:40801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 18:02:38.837993 30350 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.61:40801: Network error: Client connection negotiation failed: client connection to 127.7.118.61:40801: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.61.keytab.
Entry for principal kudu/127.7.118.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.61.keytab.
I20260929 18:02:38.839111 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.61:40801
--webserver_interface=127.7.118.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.61.keytab
--principal=kudu/127.7.118.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:35705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:42283,127.7.118.61:40801,127.7.118.60:42509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:02:38.887341 30350 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.61" port: 40801 } attempt: 1
I20260929 18:02:38.888336 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:38.887988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:40801 (local address 127.0.0.1:35114)
0929 18:02:38.888185 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:38.888252 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.61:40801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 18:02:38.889225 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:38.888981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:40801 (local address 127.0.0.1:35128)
0929 18:02:38.889103 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:38.889161 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.61:40801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 18:02:38.889369 30350 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.61:40801: Network error: Client connection negotiation failed: client connection to 127.7.118.61:40801: connect: Connection refused (error 111)
I20260929 18:02:38.957583 30350 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.61" port: 40801 } attempt: 2
I20260929 18:02:38.958599 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:38.958135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:40801 (local address 127.0.0.1:35134)
0929 18:02:38.958407 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:38.958508 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.61:40801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260929 18:02:38.959409 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:38.959197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:40801 (local address 127.0.0.1:35150)
0929 18:02:38.959301 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:38.959349 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.61:40801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 18:02:38.959602 30350 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.61:40801: Network error: Client connection negotiation failed: client connection to 127.7.118.61:40801: connect: Connection refused (error 111)
W20260929 18:02:38.962831 30357 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:02:38.963166 30357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:38.963232 30357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:38.963304 30357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:38.967886 30357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:38.968009 30357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:38.968035 30357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:38.968053 30357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:38.968103 30357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:38.974221 30357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:35705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:42283,127.7.118.61:40801,127.7.118.60:42509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.61.keytab
--principal=kudu/127.7.118.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.61:40801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.118.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:38.975550 30357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:38.976715 30357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:38.987051 30363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:38.987146 30362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:38.986944 30365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:38.987526 30357 server_base.cc:1034] running on GCE node
I20260929 18:02:38.988272 30357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:38.989616 30357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:38.990912 30357 hybrid_clock.cc:648] HybridClock initialized: now 1790704958990878 us; error 49 us; skew 500 ppm
Sep 29 18:02:38 dist-test-slave-r7ph krb5kdc[30269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704958, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:38.995008 30357 init.cc:377] Logged in from keytab as kudu/127.7.118.61@KRBTEST.COM (short username kudu)
I20260929 18:02:38.996460 30357 webserver.cc:460] Webserver started at http://127.7.118.61:41707/ using document root <none> and password file <none>
I20260929 18:02:38.997208 30357 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:38.997327 30357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:38.997613 30357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:39.000227 30357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-1/data/instance:
uuid: "1299ee17a8ae457689438596dbb843a9"
format_stamp: "Formatted at 2026-09-29 18:02:38 on dist-test-slave-r7ph"
server_key: "f1e52d643940878d7f4f507a9a624ae3"
server_key_iv: "cddd31c20a8290c8f7d7d3854fc29a1b"
server_key_version: "encryptionkey@0"
I20260929 18:02:39.000981 30357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-1/wal/instance:
uuid: "1299ee17a8ae457689438596dbb843a9"
format_stamp: "Formatted at 2026-09-29 18:02:38 on dist-test-slave-r7ph"
server_key: "f1e52d643940878d7f4f507a9a624ae3"
server_key_iv: "cddd31c20a8290c8f7d7d3854fc29a1b"
server_key_version: "encryptionkey@0"
I20260929 18:02:39.005775 30357 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 18:02:39.009135 30372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:39.011191 30357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:02:39.011386 30357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-1/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-1/wal
uuid: "1299ee17a8ae457689438596dbb843a9"
format_stamp: "Formatted at 2026-09-29 18:02:38 on dist-test-slave-r7ph"
server_key: "f1e52d643940878d7f4f507a9a624ae3"
server_key_iv: "cddd31c20a8290c8f7d7d3854fc29a1b"
server_key_version: "encryptionkey@0"
I20260929 18:02:39.011539 30357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:39.050873 30350 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.61" port: 40801 } attempt: 3
I20260929 18:02:39.052093 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.051664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:40801 (local address 127.0.0.1:35152)
0929 18:02:39.051859 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:39.051981 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.61:40801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 18:02:39.053174 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.052891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:40801 (local address 127.0.0.1:35158)
0929 18:02:39.053034 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:39.053102 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.61:40801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 18:02:39.053333 30350 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.61:40801: Network error: Client connection negotiation failed: client connection to 127.7.118.61:40801: connect: Connection refused (error 111)
I20260929 18:02:39.059147 30357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:39.063199 30357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:39.063463 30357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:39.071516 30357 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.61:40801
I20260929 18:02:39.071815 30424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.61:40801 every 8 connection(s)
I20260929 18:02:39.072733 30357 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-1/data/info.pb
I20260929 18:02:39.075894 30425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:39.078025 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 30357
I20260929 18:02:39.078161 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-1/wal/instance
I20260929 18:02:39.078504 7640 external_mini_cluster.cc:1428] Setting key dbcf074e136aada755657a50b04860c9
I20260929 18:02:39.081025 30425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42283 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.118.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.118.60@KRBTEST.COM" created.
Sep 29 18:02:39 dist-test-slave-r7ph krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704958, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.61@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:02:39.099463 30354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.083609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54866 (local address 127.7.118.62:42283)
0929 18:02:39.083755 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:39.083802 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:39.085174 (+ 1372us) server_negotiation.cc:407] Connection header received
0929 18:02:39.086317 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:39.086321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:39.086396 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:39.086502 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:39.087839 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.088400 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.089983 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.090365 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.090500 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:39.093805 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:39.093810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:39.093813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:39.093849 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:39.096389 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.097251 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.097254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.097255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.097353 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.097950 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.097953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.097954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.098184 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:39.098345 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:39.099183 (+ 838us) server_negotiation.cc:299] Negotiation successful
0929 18:02:39.099323 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:02:39.100796 30428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.084306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42283 (local address 127.0.0.1:54866)
0929 18:02:39.084983 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:39.085033 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:39.086046 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:39.086698 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:39.086706 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:39.087088 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:39.087679 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:39.087695 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.088556 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.088559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:39.089827 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:39.089839 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.090553 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.090557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:39.090759 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:39.091474 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:02:39.091503 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:39.093559 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:39.096587 (+ 3028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:39.096599 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:39.096624 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:39.097099 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:39.097491 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:39.097497 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:39.097500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:39.097807 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:39.098363 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:39.098373 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:39.098823 (+ 450us) client_negotiation.cc:769] Sending connection context
0929 18:02:39.099084 (+ 261us) client_negotiation.cc:240] Negotiation successful
0929 18:02:39.099535 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":220,"threads_started":1}
I20260929 18:02:39.102499 30425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.61" port: 40801 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:02:39 dist-test-slave-r7ph krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704958, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.61@KRBTEST.COM for kudu/127.7.118.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.118.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.60.keytab.
Entry for principal kudu/127.7.118.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.60.keytab.
I20260929 18:02:39.110180 7640 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8_cVK5/build/debug/bin/kudu
/tmp/dist-test-task8_cVK5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.60:42509
--webserver_interface=127.7.118.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.60.keytab
--principal=kudu/127.7.118.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.118.20:35705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.118.62:42283,127.7.118.61:40801,127.7.118.60:42509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:02:39.112687 30428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.103283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:40801 (local address 127.0.0.1:35172)
0929 18:02:39.103498 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:39.103512 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:39.103593 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:39.104202 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:39.104204 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:39.104424 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:39.104510 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:39.104515 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.105682 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.105688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:39.106313 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:39.106323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.106952 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.106955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:39.107050 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:39.107561 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:02:39.107573 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:39.108906 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:39.111069 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:39.111075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:39.111077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:39.111269 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:39.111556 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:39.111561 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:39.111563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:39.111642 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:39.112133 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:39.112136 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:39.112222 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 18:02:39.112342 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 18:02:39.112494 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260929 18:02:39.112703 30430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.103457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.7.118.61:40801)
0929 18:02:39.103826 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:39.103838 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:39.103860 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:02:39.103907 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:39.103911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:39.104082 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:39.104231 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:39.104619 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.105574 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.106427 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.106816 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.106950 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:39.109054 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:39.109064 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:39.109078 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:39.109097 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:39.110953 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.111380 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.111393 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.111395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.111464 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.111739 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.111742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.111743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.112024 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:39.112173 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:39.112433 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 18:02:39.112565 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":133,"threads_started":1}
I20260929 18:02:39.115720 30425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42509 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:02:39.116775 30428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.116395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42509 (local address 127.0.0.1:45210)
0929 18:02:39.116587 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:39.116689 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260929 18:02:39.117019 30376 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42509: connect: Connection refused (error 111)
I20260929 18:02:39.117985 30428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.117758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42509 (local address 127.0.0.1:45220)
0929 18:02:39.117873 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:39.117927 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 18:02:39.118157 30425 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.60:42509: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42509: connect: Connection refused (error 111)
I20260929 18:02:39.167529 30425 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42509 } attempt: 1
I20260929 18:02:39.168466 30428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.168126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42509 (local address 127.0.0.1:45228)
0929 18:02:39.168274 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:39.168355 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 18:02:39.169368 30428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.169036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42509 (local address 127.0.0.1:45234)
0929 18:02:39.169213 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:39.169280 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 18:02:39.169548 30425 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.60:42509: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42509: connect: Connection refused (error 111)
I20260929 18:02:39.196503 30350 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.61" port: 40801 } attempt: 4
Sep 29 18:02:39 dist-test-slave-r7ph krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704958, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for kudu/127.7.118.61@KRBTEST.COM
I20260929 18:02:39.214648 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.202644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:40801 (local address 127.0.0.1:35186)
0929 18:02:39.202844 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:39.202864 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:39.202981 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:39.203480 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:39.203484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:39.203689 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:39.203881 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:39.203902 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.205620 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.205624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:39.206702 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:39.206720 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.207530 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.207535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:39.207661 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:39.208488 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:02:39.208505 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:39.210168 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:39.212761 (+ 2593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:39.212766 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:39.212768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:39.213010 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:39.213414 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:39.213422 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:39.213424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:39.213546 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:39.214047 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:39.214050 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:39.214156 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 18:02:39.214311 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 18:02:39.214502 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 18:02:39.214664 30430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.203052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35186 (local address 127.7.118.61:40801)
0929 18:02:39.203207 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:39.203213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:39.203231 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:02:39.203281 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:39.203283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:39.203334 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:39.204032 (+ 698us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:39.204155 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.205421 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.206933 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.207355 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.208076 (+ 721us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:39.210632 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:39.210637 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:39.210640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:39.210674 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:39.212581 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.213129 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.213133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.213135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.213239 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.213658 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.213661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.213662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.213933 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:39.214037 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:39.214366 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 18:02:39.214512 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:02:39.215862 30350 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42509 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:02:39.216657 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.216388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42509 (local address 127.0.0.1:45240)
0929 18:02:39.216526 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:39.216582 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 18:02:39.217540 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.217274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42509 (local address 127.0.0.1:45256)
0929 18:02:39.217406 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:39.217470 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 18:02:39.217764 30350 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.60:42509: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42509: connect: Connection refused (error 111)
I20260929 18:02:39.237816 30425 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42509 } attempt: 2
I20260929 18:02:39.238739 30428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.238444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42509 (local address 127.0.0.1:45268)
0929 18:02:39.238589 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:39.238661 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 18:02:39.239585 30428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.239328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42509 (local address 127.0.0.1:45280)
0929 18:02:39.239449 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:39.239504 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 18:02:39.239771 30425 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.60:42509: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42509: connect: Connection refused (error 111)
W20260929 18:02:39.240396 30431 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:02:39.240685 30431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:02:39.240710 30431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:02:39.240801 30431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:02:39.245644 30431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:02:39.245729 30431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:02:39.245751 30431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:02:39.245769 30431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:02:39.245785 30431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:02:39.251688 30431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.118.20:35705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.118.62:42283,127.7.118.61:40801,127.7.118.60:42509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/krb5kdc/kudu_127.7.118.60.keytab
--principal=kudu/127.7.118.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.118.60:42509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.118.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 18:02:39.252955 30431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:02:39.254066 30431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:02:39.263294 30440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:39.263518 30431 server_base.cc:1034] running on GCE node
W20260929 18:02:39.263284 30438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:02:39.263284 30437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:02:39.264305 30431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:02:39.265625 30431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:02:39.266994 30431 hybrid_clock.cc:648] HybridClock initialized: now 1790704959266979 us; error 31 us; skew 500 ppm
Sep 29 18:02:39 dist-test-slave-r7ph krb5kdc[30269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704959, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:02:39.270186 30431 init.cc:377] Logged in from keytab as kudu/127.7.118.60@KRBTEST.COM (short username kudu)
I20260929 18:02:39.271265 30431 webserver.cc:460] Webserver started at http://127.7.118.60:42831/ using document root <none> and password file <none>
I20260929 18:02:39.271883 30431 fs_manager.cc:359] Metadata directory not provided
I20260929 18:02:39.271979 30431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:02:39.272230 30431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:02:39.274466 30431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-2/data/instance:
uuid: "41a8dc8657cb42cd80d0e47339f45909"
format_stamp: "Formatted at 2026-09-29 18:02:39 on dist-test-slave-r7ph"
server_key: "5da2af6c3ade5effb8e8b12dfd132165"
server_key_iv: "818f92b2c773467b216cd3b0d226046f"
server_key_version: "encryptionkey@0"
I20260929 18:02:39.275089 30431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-2/wal/instance:
uuid: "41a8dc8657cb42cd80d0e47339f45909"
format_stamp: "Formatted at 2026-09-29 18:02:39 on dist-test-slave-r7ph"
server_key: "5da2af6c3ade5effb8e8b12dfd132165"
server_key_iv: "818f92b2c773467b216cd3b0d226046f"
server_key_version: "encryptionkey@0"
I20260929 18:02:39.277033 30350 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42509 } attempt: 1
I20260929 18:02:39.277894 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.277562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42509 (local address 127.0.0.1:45294)
0929 18:02:39.277737 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:39.277806 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 18:02:39.278780 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.278537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42509 (local address 127.0.0.1:45304)
0929 18:02:39.278654 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:39.278712 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 18:02:39.278949 30350 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.118.60:42509: Network error: Client connection negotiation failed: client connection to 127.7.118.60:42509: connect: Connection refused (error 111)
I20260929 18:02:39.279467 30431 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 18:02:39.282117 30447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:02:39.283547 30431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:02:39.283707 30431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-2/data,/tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-2/wal
uuid: "41a8dc8657cb42cd80d0e47339f45909"
format_stamp: "Formatted at 2026-09-29 18:02:39 on dist-test-slave-r7ph"
server_key: "5da2af6c3ade5effb8e8b12dfd132165"
server_key_iv: "818f92b2c773467b216cd3b0d226046f"
server_key_version: "encryptionkey@0"
I20260929 18:02:39.283849 30431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:02:39.299903 30431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:02:39.303768 30431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:02:39.304023 30431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:02:39.311823 30431 rpc_server.cc:307] RPC server started. Bound to: 127.7.118.60:42509
I20260929 18:02:39.312182 30499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.118.60:42509 every 8 connection(s)
I20260929 18:02:39.312911 30431 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-2/data/info.pb
I20260929 18:02:39.315519 7640 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8_cVK5/build/debug/bin/kudu as pid 30431
I20260929 18:02:39.315702 7640 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8_cVK5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704657566865-7640-0/minicluster-data/master-2/wal/instance
I20260929 18:02:39.316040 7640 external_mini_cluster.cc:1428] Setting key 7788854610f474d592c29b07d7390b4f
I20260929 18:02:39.316395 30500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:02:39.321729 30500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42283 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:02:39 dist-test-slave-r7ph krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.60@KRBTEST.COM
Sep 29 18:02:39 dist-test-slave-r7ph krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704959, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.60@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
I20260929 18:02:39.330976 30425 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42509 } attempt: 3
I20260929 18:02:39.333350 30503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.317480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45314 (local address 127.7.118.60:42509)
0929 18:02:39.317905 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:39.317922 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:39.317960 (+ 38us) server_negotiation.cc:407] Connection header received
0929 18:02:39.318766 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:39.318795 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:39.319170 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:39.319476 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:39.320476 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.321540 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.322807 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.323358 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.323578 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:39.326775 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:39.326780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:39.326790 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:39.326820 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:39.329034 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.329603 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.329612 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.329619 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.329801 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.330235 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.330238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.330239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.330758 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:39.331241 (+ 483us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:39.331452 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 18:02:39.332104 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":137,"threads_started":1}
I20260929 18:02:39.334547 30504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.323929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.62:42283 (local address 127.0.0.1:54876)
0929 18:02:39.324479 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:39.324541 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:39.324688 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:39.325039 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:39.325048 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:39.325550 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:39.325694 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:39.325701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.326684 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.326690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:39.327423 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:39.327434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.327902 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.327906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:39.328036 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:39.328573 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:02:39.328586 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:39.330493 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:39.332551 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:39.332558 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:39.332565 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:39.332838 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:39.333226 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:39.333237 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:39.333239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:39.333343 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:39.333779 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:39.333785 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:39.333921 (+ 136us) client_negotiation.cc:769] Sending connection context
0929 18:02:39.334125 (+ 204us) client_negotiation.cc:240] Negotiation successful
0929 18:02:39.334347 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":125,"threads_started":1}
I20260929 18:02:39.335044 30354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.323954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54876 (local address 127.7.118.62:42283)
0929 18:02:39.324109 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:39.324115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:39.324655 (+ 540us) server_negotiation.cc:407] Connection header received
0929 18:02:39.324809 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:39.324812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:39.324862 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:39.324933 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:39.325856 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.326544 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.327558 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.327784 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.327902 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:39.330657 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:39.330660 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:39.330662 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:39.330691 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:39.332446 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.333015 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.333019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.333020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.333120 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.333452 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.333455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.333457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.333677 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:39.333774 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:39.334240 (+ 466us) server_negotiation.cc:299] Negotiation successful
0929 18:02:39.334416 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:02:39.336642 30500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.61" port: 40801 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:02:39 dist-test-slave-r7ph krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704958, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.61@KRBTEST.COM for kudu/127.7.118.60@KRBTEST.COM
I20260929 18:02:39.341566 30428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.331461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42509 (local address 127.0.0.1:45322)
0929 18:02:39.331777 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:39.331796 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:39.331877 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:39.332474 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:39.332480 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:39.332668 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:39.332768 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:39.332773 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.333818 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.333823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:39.334699 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:39.334713 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.335344 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.335347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:39.335457 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:39.335986 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:02:39.336013 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:39.337615 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:39.339864 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:39.339867 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:39.339869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:39.340135 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:39.340453 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:39.340456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:39.340458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:39.340555 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:39.341002 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:39.341005 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:39.341115 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 18:02:39.341229 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 18:02:39.341393 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20260929 18:02:39.341598 30505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.331806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45322 (local address 127.7.118.60:42509)
0929 18:02:39.332221 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:39.332229 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:39.332242 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:02:39.332277 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:39.332280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:39.332322 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:39.332409 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:39.332893 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.333656 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.334862 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.335228 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.335328 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:39.337766 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:39.337769 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:39.337773 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:39.337803 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:39.339737 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.340261 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.340264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.340266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.340353 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.340658 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.340660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.340662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.340894 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:39.341007 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:39.341282 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 18:02:39.341398 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":188,"threads_started":1}
Sep 29 18:02:39 dist-test-slave-r7ph krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.62@KRBTEST.COM
Sep 29 18:02:39 dist-test-slave-r7ph krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704959, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.60@KRBTEST.COM for kudu/127.7.118.61@KRBTEST.COM
I20260929 18:02:39.346002 30425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1299ee17a8ae457689438596dbb843a9: Bootstrap starting.
I20260929 18:02:39.346307 30350 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42509 } attempt: 2
I20260929 18:02:39.348151 30354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.336115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54880 (local address 127.7.118.62:42283)
0929 18:02:39.336324 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:39.336328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:39.336506 (+ 178us) server_negotiation.cc:407] Connection header received
0929 18:02:39.336690 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:39.336693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:39.336747 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:39.336825 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:39.337688 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.338446 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.339448 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.339686 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.339798 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:39.342741 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:39.342748 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:39.342753 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:39.342796 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:39.344646 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.346053 (+ 1407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.346059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.346061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.346161 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.346544 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.346548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.346550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.346833 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:39.347105 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:39.347841 (+ 736us) server_negotiation.cc:299] Negotiation successful
0929 18:02:39.347984 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260929 18:02:39.348439 30430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.337416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35188 (local address 127.7.118.61:40801)
0929 18:02:39.337533 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:39.337539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:39.337551 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:02:39.337590 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:39.337593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:39.337630 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:39.337698 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:39.338866 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.339719 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.340842 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.341257 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.341385 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:39.343902 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:39.343906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:39.343909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:39.343934 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:39.345576 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.346104 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.346108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.346110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.346192 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.347404 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.347408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.347409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.347699 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:39.347785 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:39.348164 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 18:02:39.348302 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:02:39.349443 30504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.337167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.61:40801 (local address 127.0.0.1:35188)
0929 18:02:39.337330 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:39.337346 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:39.337444 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:39.337857 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:39.337863 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:39.338050 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:39.338693 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:39.338707 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.339859 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.339862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:39.340710 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:39.340717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.341531 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.341535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:39.341670 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:39.342179 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:02:39.342194 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:39.343744 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:39.345706 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:39.345709 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:39.345711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:39.345912 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:39.347162 (+ 1250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:39.347165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:39.347167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:39.347279 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:39.347864 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:39.347867 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:39.347973 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 18:02:39.348921 (+ 948us) client_negotiation.cc:240] Negotiation successful
0929 18:02:39.349287 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 18:02:39.349982 30425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1299ee17a8ae457689438596dbb843a9: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:39.350847 30500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42509 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:02:39.351120 30425 log.cc:824] T 00000000000000000000000000000000 P 1299ee17a8ae457689438596dbb843a9: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:02:39 dist-test-slave-r7ph krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704958, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.62@KRBTEST.COM for kudu/127.7.118.60@KRBTEST.COM
I20260929 18:02:39.355603 30425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1299ee17a8ae457689438596dbb843a9: No bootstrap required, opened a new log
Sep 29 18:02:39 dist-test-slave-r7ph krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704959, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.118.60@KRBTEST.COM for kudu/127.7.118.60@KRBTEST.COM
I20260929 18:02:39.357151 30505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.346811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45328 (local address 127.7.118.60:42509)
0929 18:02:39.346958 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:39.346963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:39.346977 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:02:39.347116 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:39.347120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:39.347172 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:39.347840 (+ 668us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:39.347957 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.348986 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.350293 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.350685 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.351267 (+ 582us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:39.353291 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:39.353295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:39.353298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:39.353323 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:39.355140 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.355737 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.355740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.355742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.355834 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.356184 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.356187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.356188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.356404 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:39.356506 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:39.356859 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 18:02:39.356993 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:02:39.357156 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.346729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42509 (local address 127.0.0.1:45328)
0929 18:02:39.346925 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:39.346940 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:39.347014 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:39.347304 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:39.347308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:39.347500 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:39.347620 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:39.347627 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.349119 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.349124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:39.349963 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:39.349977 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.350823 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.350827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:39.350921 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:39.351587 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:02:39.351602 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:39.353138 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:39.355280 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:39.355285 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:39.355289 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:39.355603 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:39.355951 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:39.355955 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:39.355957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:39.356054 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:39.356530 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:39.356536 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:39.356653 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 18:02:39.356780 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 18:02:39.356952 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
Sep 29 18:02:39 dist-test-slave-r7ph krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.118.61@KRBTEST.COM
I20260929 18:02:39.360749 30425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1299ee17a8ae457689438596dbb843a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bc9605c80e41a7b6eccf67f530977c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42283 } } peers { permanent_uuid: "1299ee17a8ae457689438596dbb843a9" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 40801 } } peers { permanent_uuid: "41a8dc8657cb42cd80d0e47339f45909" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42509 } }
I20260929 18:02:39.361068 30425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1299ee17a8ae457689438596dbb843a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1299ee17a8ae457689438596dbb843a9, State: Initialized, Role: FOLLOWER
I20260929 18:02:39.361105 30350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c: Bootstrap starting.
I20260929 18:02:39.361462 30425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1299ee17a8ae457689438596dbb843a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bc9605c80e41a7b6eccf67f530977c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42283 } } peers { permanent_uuid: "1299ee17a8ae457689438596dbb843a9" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 40801 } } peers { permanent_uuid: "41a8dc8657cb42cd80d0e47339f45909" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42509 } }
I20260929 18:02:39.362035 30507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1299ee17a8ae457689438596dbb843a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bc9605c80e41a7b6eccf67f530977c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42283 } } peers { permanent_uuid: "1299ee17a8ae457689438596dbb843a9" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 40801 } } peers { permanent_uuid: "41a8dc8657cb42cd80d0e47339f45909" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42509 } } }
I20260929 18:02:39.362313 30430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.349807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35204 (local address 127.7.118.61:40801)
0929 18:02:39.349922 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:39.349926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:39.349995 (+ 69us) server_negotiation.cc:407] Connection header received
0929 18:02:39.350427 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:39.350431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:39.350474 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:39.350547 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:39.351572 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.352325 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.353472 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.353814 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.353885 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:39.358390 (+ 4505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:39.358394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:39.358397 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:39.358426 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:39.360100 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.360650 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.360654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.360656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.360761 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.361121 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.361125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.361130 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.361370 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:39.361562 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:39.361995 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 18:02:39.362132 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:02:39.362299 30507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1299ee17a8ae457689438596dbb843a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:02:39.363180 30425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1299ee17a8ae457689438596dbb843a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:02:39.365097 30350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:39.366005 30504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.351382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.118.60:42509 (local address 127.0.0.1:45332)
0929 18:02:39.352133 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0929 18:02:39.352148 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:02:39.352249 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:02:39.352544 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:02:39.352546 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:02:39.352719 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:02:39.352816 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:39.352822 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.353788 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.353791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:39.354515 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:02:39.354525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.355031 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.355034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:02:39.355109 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:39.355564 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:02:39.355585 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:02:39.357669 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:02:39.364175 (+ 6506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:39.364178 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:39.364180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:39.364458 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:39.364830 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:02:39.364832 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:02:39.364834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:02:39.364920 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:02:39.365451 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:02:39.365454 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:02:39.365549 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 18:02:39.365669 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 18:02:39.365839 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679}
I20260929 18:02:39.366799 7640 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:02:39.366891 30503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.351581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45332 (local address 127.7.118.60:42509)
0929 18:02:39.351699 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:39.351781 (+ 82us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:39.352198 (+ 417us) server_negotiation.cc:407] Connection header received
0929 18:02:39.352331 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:39.352334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:39.352379 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:39.352426 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:39.352929 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.353668 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.354667 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.354905 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.355030 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:39.357843 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:39.357847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:39.357849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:39.357873 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:39.364027 (+ 6154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.364639 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.364642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.364644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.364725 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.365013 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.365017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.365021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.365241 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:39.365346 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:39.366616 (+ 1270us) server_negotiation.cc:299] Negotiation successful
0929 18:02:39.366738 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 18:02:39.368038 30324 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "10bc9605c80e41a7b6eccf67f530977c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54880
W20260929 18:02:39.369561 30518 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1299ee17a8ae457689438596dbb843a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:02:39.369680 30518 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:02:39.370146 30500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41a8dc8657cb42cd80d0e47339f45909: Bootstrap starting.
I20260929 18:02:39.373476 30500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41a8dc8657cb42cd80d0e47339f45909: Neither blocks nor log segments found. Creating new log.
I20260929 18:02:39.374338 30357 master_runner.cc:186] Error getting master registration for 127.7.118.62:42283: OK, instance_id { permanent_uuid: "10bc9605c80e41a7b6eccf67f530977c" instance_seqno: 1790704958800201 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:02:39.378000 30357 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:02:39.378479 30500 log.cc:824] T 00000000000000000000000000000000 P 41a8dc8657cb42cd80d0e47339f45909: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:39.379478 30350 log.cc:824] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:02:39.381299 30500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41a8dc8657cb42cd80d0e47339f45909: No bootstrap required, opened a new log
I20260929 18:02:39.381708 30350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c: No bootstrap required, opened a new log
I20260929 18:02:39.385031 30350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bc9605c80e41a7b6eccf67f530977c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42283 } } peers { permanent_uuid: "1299ee17a8ae457689438596dbb843a9" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 40801 } } peers { permanent_uuid: "41a8dc8657cb42cd80d0e47339f45909" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42509 } }
I20260929 18:02:39.385304 30350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bc9605c80e41a7b6eccf67f530977c, State: Initialized, Role: FOLLOWER
I20260929 18:02:39.385677 30350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bc9605c80e41a7b6eccf67f530977c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42283 } } peers { permanent_uuid: "1299ee17a8ae457689438596dbb843a9" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 40801 } } peers { permanent_uuid: "41a8dc8657cb42cd80d0e47339f45909" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42509 } }
I20260929 18:02:39.385594 30500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41a8dc8657cb42cd80d0e47339f45909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bc9605c80e41a7b6eccf67f530977c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42283 } } peers { permanent_uuid: "1299ee17a8ae457689438596dbb843a9" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 40801 } } peers { permanent_uuid: "41a8dc8657cb42cd80d0e47339f45909" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42509 } }
I20260929 18:02:39.385852 30500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41a8dc8657cb42cd80d0e47339f45909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a8dc8657cb42cd80d0e47339f45909, State: Initialized, Role: FOLLOWER
I20260929 18:02:39.386169 30500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41a8dc8657cb42cd80d0e47339f45909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bc9605c80e41a7b6eccf67f530977c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42283 } } peers { permanent_uuid: "1299ee17a8ae457689438596dbb843a9" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 40801 } } peers { permanent_uuid: "41a8dc8657cb42cd80d0e47339f45909" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42509 } }
I20260929 18:02:39.386317 30522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bc9605c80e41a7b6eccf67f530977c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42283 } } peers { permanent_uuid: "1299ee17a8ae457689438596dbb843a9" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 40801 } } peers { permanent_uuid: "41a8dc8657cb42cd80d0e47339f45909" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42509 } } }
I20260929 18:02:39.386675 30522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:02:39.387328 30350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:02:39.387372 30324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 18:02:39.387467 30324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:39.387552 30500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41a8dc8657cb42cd80d0e47339f45909 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:02:39.389050 30324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bc9605c80e41a7b6eccf67f530977c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42283 } } peers { permanent_uuid: "1299ee17a8ae457689438596dbb843a9" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 40801 } } peers { permanent_uuid: "41a8dc8657cb42cd80d0e47339f45909" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42509 } }
I20260929 18:02:39.390213 30523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41a8dc8657cb42cd80d0e47339f45909 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bc9605c80e41a7b6eccf67f530977c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42283 } } peers { permanent_uuid: "1299ee17a8ae457689438596dbb843a9" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 40801 } } peers { permanent_uuid: "41a8dc8657cb42cd80d0e47339f45909" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42509 } } }
I20260929 18:02:39.390410 30523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41a8dc8657cb42cd80d0e47339f45909 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:02:39.391176 30399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "10bc9605c80e41a7b6eccf67f530977c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1299ee17a8ae457689438596dbb843a9"
I20260929 18:02:39.391417 30474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "10bc9605c80e41a7b6eccf67f530977c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "41a8dc8657cb42cd80d0e47339f45909"
I20260929 18:02:39.391505 30399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1299ee17a8ae457689438596dbb843a9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:39.391611 30474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41a8dc8657cb42cd80d0e47339f45909 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:02:39.392994 30474 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 41a8dc8657cb42cd80d0e47339f45909 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bc9605c80e41a7b6eccf67f530977c in term 1.
I20260929 18:02:39.392994 30399 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1299ee17a8ae457689438596dbb843a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bc9605c80e41a7b6eccf67f530977c in term 1.
W20260929 18:02:39.393584 30545 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 41a8dc8657cb42cd80d0e47339f45909: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:02:39.393661 30545 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:02:39.390209 30324 leader_election.cc:288] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c [CANDIDATE]: Term 1 election: Requested vote from peers 1299ee17a8ae457689438596dbb843a9 (127.7.118.61:40801), 41a8dc8657cb42cd80d0e47339f45909 (127.7.118.60:42509)
I20260929 18:02:39.393510 30300 leader_election.cc:302] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10bc9605c80e41a7b6eccf67f530977c, 1299ee17a8ae457689438596dbb843a9; no voters:
I20260929 18:02:39.398059 30522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:02:39.398432 30522 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c [term 1 LEADER]: Becoming Leader. State: Replica: 10bc9605c80e41a7b6eccf67f530977c, State: Running, Role: LEADER
I20260929 18:02:39.398825 30522 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bc9605c80e41a7b6eccf67f530977c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42283 } } peers { permanent_uuid: "1299ee17a8ae457689438596dbb843a9" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 40801 } } peers { permanent_uuid: "41a8dc8657cb42cd80d0e47339f45909" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42509 } }
I20260929 18:02:39.400941 30431 master_runner.cc:263] No leader master found from master 41a8dc8657cb42cd80d0e47339f45909
I20260929 18:02:39.401011 30431 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:02:39.401786 30522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10bc9605c80e41a7b6eccf67f530977c. Latest consensus state: current_term: 1 leader_uuid: "10bc9605c80e41a7b6eccf67f530977c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bc9605c80e41a7b6eccf67f530977c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42283 } } peers { permanent_uuid: "1299ee17a8ae457689438596dbb843a9" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 40801 } } peers { permanent_uuid: "41a8dc8657cb42cd80d0e47339f45909" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42509 } } }
I20260929 18:02:39.401918 30522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:39.402345 30282 master_runner.cc:263] No leader master found from master 10bc9605c80e41a7b6eccf67f530977c
I20260929 18:02:39.402417 30282 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 18:02:39.412499 30544 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:02:39.412603 30544 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:02:39.412691 30430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.397701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35212 (local address 127.7.118.61:40801)
0929 18:02:39.397823 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:39.397827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:39.402554 (+ 4727us) server_negotiation.cc:407] Connection header received
0929 18:02:39.402771 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:39.402774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:39.402825 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:39.402888 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:39.404070 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.404898 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.406238 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.406593 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.406694 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:39.408716 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:39.408720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:39.408723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:39.408756 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:39.410568 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.411108 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.411113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.411116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.411218 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.411581 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.411585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.411587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.411857 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:39.411980 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:39.412359 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 18:02:39.412505 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:02:39.412943 30554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:02:39.414023 30554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:02:39.426056 30503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.415742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45338 (local address 127.7.118.60:42509)
0929 18:02:39.415896 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:39.415901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:39.415919 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:02:39.415965 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:39.415968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:39.416022 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:39.416101 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:39.416987 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.417762 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.419276 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.419646 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.419738 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:39.421793 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:39.421797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:39.421800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:39.421833 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:39.423792 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.424419 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.424425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.424427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.424557 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.424919 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.424923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.424925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.425168 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:39.425297 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:39.425694 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 18:02:39.425837 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 18:02:39.432549 30354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:39.410676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54892 (local address 127.7.118.62:42283)
0929 18:02:39.410825 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:39.410830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:39.414630 (+ 3800us) server_negotiation.cc:407] Connection header received
0929 18:02:39.414882 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:39.414885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:39.414941 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:39.414991 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:39.416026 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.416851 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.418100 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:39.418459 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:39.418541 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:39.427594 (+ 9053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:02:39.427598 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:02:39.427601 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:02:39.427631 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:02:39.430085 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.430717 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.430721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.430723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.430829 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:02:39.431133 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:02:39.431136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:02:39.431138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:02:39.431390 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:02:39.431514 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:39.432242 (+ 728us) server_negotiation.cc:299] Negotiation successful
0929 18:02:39.432397 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:02:39.433568 30474 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 41a8dc8657cb42cd80d0e47339f45909 [term 1 FOLLOWER]: Refusing update from remote peer 10bc9605c80e41a7b6eccf67f530977c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:02:39.434563 30552 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c [LEADER]: Connected to new peer: Peer: permanent_uuid: "41a8dc8657cb42cd80d0e47339f45909" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:02:39.436200 30398 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1299ee17a8ae457689438596dbb843a9 [term 1 FOLLOWER]: Refusing update from remote peer 10bc9605c80e41a7b6eccf67f530977c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:02:39.438907 30552 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1299ee17a8ae457689438596dbb843a9" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 40801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:02:39.444844 30507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1299ee17a8ae457689438596dbb843a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10bc9605c80e41a7b6eccf67f530977c. Latest consensus state: current_term: 1 leader_uuid: "10bc9605c80e41a7b6eccf67f530977c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bc9605c80e41a7b6eccf67f530977c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42283 } } peers { permanent_uuid: "1299ee17a8ae457689438596dbb843a9" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 40801 } } peers { permanent_uuid: "41a8dc8657cb42cd80d0e47339f45909" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42509 } } }
I20260929 18:02:39.445024 30507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1299ee17a8ae457689438596dbb843a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:02:39.447152 30523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41a8dc8657cb42cd80d0e47339f45909 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10bc9605c80e41a7b6eccf67f530977c. Latest consensus state: current_term: 1 leader_uuid: "10bc9605c80e41a7b6eccf67f530977c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bc9605c80e41a7b6eccf67f530977c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42283 } } peers { permanent_uuid: "1299ee17a8ae457689438596dbb843a9" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 40801 } } peers { permanent_uuid: "41a8dc8657cb42cd80d0e47339f45909" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42509 } } }
I20260929 18:02:39.447324 30523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41a8dc8657cb42cd80d0e47339f45909 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:02:39.449564 30552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "10bc9605c80e41a7b6eccf67f530977c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bc9605c80e41a7b6eccf67f530977c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42283 } } peers { permanent_uuid: "1299ee17a8ae457689438596dbb843a9" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 40801 } } peers { permanent_uuid: "41a8dc8657cb42cd80d0e47339f45909" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42509 } } }
I20260929 18:02:39.449719 30552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:39.450232 30558 mvcc.cc:205] Tried to move back new op lower bound from 7334727513819455488 to 7334727513705144320. Current Snapshot: MvccSnapshot[applied={T|T < 7334727513819455488}]
I20260929 18:02:39.451026 30507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1299ee17a8ae457689438596dbb843a9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "10bc9605c80e41a7b6eccf67f530977c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bc9605c80e41a7b6eccf67f530977c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42283 } } peers { permanent_uuid: "1299ee17a8ae457689438596dbb843a9" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 40801 } } peers { permanent_uuid: "41a8dc8657cb42cd80d0e47339f45909" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42509 } } }
I20260929 18:02:39.451156 30507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1299ee17a8ae457689438596dbb843a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:02:39.451175 30560 mvcc.cc:205] Tried to move back new op lower bound from 7334727513819455488 to 7334727513705144320. Current Snapshot: MvccSnapshot[applied={T|T < 7334727513819455488}]
I20260929 18:02:39.454277 30522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "10bc9605c80e41a7b6eccf67f530977c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bc9605c80e41a7b6eccf67f530977c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42283 } } peers { permanent_uuid: "1299ee17a8ae457689438596dbb843a9" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 40801 } } peers { permanent_uuid: "41a8dc8657cb42cd80d0e47339f45909" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42509 } } }
I20260929 18:02:39.454438 30522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c [sys.catalog]: This master's current role is: LEADER
I20260929 18:02:39.455824 30523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41a8dc8657cb42cd80d0e47339f45909 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "10bc9605c80e41a7b6eccf67f530977c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bc9605c80e41a7b6eccf67f530977c" member_type: VOTER last_known_addr { host: "127.7.118.62" port: 42283 } } peers { permanent_uuid: "1299ee17a8ae457689438596dbb843a9" member_type: VOTER last_known_addr { host: "127.7.118.61" port: 40801 } } peers { permanent_uuid: "41a8dc8657cb42cd80d0e47339f45909" member_type: VOTER last_known_addr { host: "127.7.118.60" port: 42509 } } }
I20260929 18:02:39.455974 30523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41a8dc8657cb42cd80d0e47339f45909 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:02:39.463510 30554 catalog_manager.cc:1347] Generated new cluster ID: c6654507bd86423b94e3e5c4bbaaebe7
I20260929 18:02:39.463619 30554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:02:39.492810 30554 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:02:39.494375 30554 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:02:39.528297 30554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10bc9605c80e41a7b6eccf67f530977c: Generated new TSK 0
I20260929 18:02:39.529320 30554 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:02:39.568150 30503 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:02:39.564891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45346 (local address 127.7.118.60:42509)
0929 18:02:39.565045 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:39.565050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:39.566951 (+ 1901us) server_negotiation.cc:407] Connection header received
0929 18:02:39.567093 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:39.567098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:39.567159 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:39.567247 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:39.567997 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45346: BlockingRecv error: recv got EOF from 127.0.0.1:45346 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 18:02:39.569142 30430 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:02:39.565708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35228 (local address 127.7.118.61:40801)
0929 18:02:39.565821 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:39.565826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:39.567368 (+ 1542us) server_negotiation.cc:407] Connection header received
0929 18:02:39.567436 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:39.567438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:39.567493 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:39.567572 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:39.569017 (+ 1445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35228: BlockingRecv error: recv got EOF from 127.0.0.1:35228 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260929 18:02:39.569144 30354 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:02:39.566393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54906 (local address 127.7.118.62:42283)
0929 18:02:39.566516 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:39.566520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:39.568216 (+ 1696us) server_negotiation.cc:407] Connection header received
0929 18:02:39.568407 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:39.568412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:39.568473 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:39.568524 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:39.569016 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54906: BlockingRecv error: recv got EOF from 127.0.0.1:54906 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 18:02:39.574491 30430 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:02:39.572727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35244 (local address 127.7.118.61:40801)
0929 18:02:39.572947 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:39.572954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:39.573331 (+ 377us) server_negotiation.cc:407] Connection header received
0929 18:02:39.573593 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:39.573597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:39.573682 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:39.573738 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:39.574365 (+ 627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35244: BlockingRecv error: recv got EOF from 127.0.0.1:35244 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260929 18:02:39.580168 30430 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:02:39.578409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35248 (local address 127.7.118.61:40801)
0929 18:02:39.578537 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:39.578542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:39.579137 (+ 595us) server_negotiation.cc:407] Connection header received
0929 18:02:39.579348 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:39.579352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:39.579407 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:39.579462 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:39.580042 (+ 580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35248: BlockingRecv error: recv got EOF from 127.0.0.1:35248 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 18:02:39.587143 30430 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:02:39.585062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35250 (local address 127.7.118.61:40801)
0929 18:02:39.585203 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:39.585208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:39.585920 (+ 712us) server_negotiation.cc:407] Connection header received
0929 18:02:39.586146 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:39.586150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:39.586210 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:39.586371 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:39.586997 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35250: BlockingRecv error: recv got EOF from 127.0.0.1:35250 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 18:02:39.595453 30430 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:02:39.593952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.7.118.61:40801)
0929 18:02:39.594146 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:39.594151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:39.594385 (+ 234us) server_negotiation.cc:407] Connection header received
0929 18:02:39.594537 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:39.594541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:39.594615 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:39.594730 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:39.595313 (+ 583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35264: BlockingRecv error: recv got EOF from 127.0.0.1:35264 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
W20260929 18:02:39.607314 30430 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:02:39.605902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35272 (local address 127.7.118.61:40801)
0929 18:02:39.606039 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:39.606043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:39.606312 (+ 269us) server_negotiation.cc:407] Connection header received
0929 18:02:39.606463 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:39.606467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:39.606536 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:39.606644 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:39.607191 (+ 547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35272: BlockingRecv error: recv got EOF from 127.0.0.1:35272 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 18:02:39.627028 30430 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:02:39.625620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35274 (local address 127.7.118.61:40801)
0929 18:02:39.625752 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:39.625769 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:39.626086 (+ 317us) server_negotiation.cc:407] Connection header received
0929 18:02:39.626239 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:39.626243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:39.626335 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:39.626437 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:39.626905 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35274: BlockingRecv error: recv got EOF from 127.0.0.1:35274 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 18:02:39.662683 30430 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:02:39.661350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35286 (local address 127.7.118.61:40801)
0929 18:02:39.661502 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:39.661507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:39.661657 (+ 150us) server_negotiation.cc:407] Connection header received
0929 18:02:39.661778 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:39.661781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:39.661842 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:39.661949 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:39.662558 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35286: BlockingRecv error: recv got EOF from 127.0.0.1:35286 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 18:02:39.730400 30430 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:02:39.729034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35296 (local address 127.7.118.61:40801)
0929 18:02:39.729227 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:39.729232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:39.729400 (+ 168us) server_negotiation.cc:407] Connection header received
0929 18:02:39.729580 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:39.729584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:39.729643 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:39.729737 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:39.730228 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35296: BlockingRecv error: recv got EOF from 127.0.0.1:35296 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260929 18:02:39.862339 30430 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:02:39.860886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35312 (local address 127.7.118.61:40801)
0929 18:02:39.861154 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:39.861173 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:39.861302 (+ 129us) server_negotiation.cc:407] Connection header received
0929 18:02:39.861506 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:39.861510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:39.861575 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:39.861675 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:39.862178 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35312: BlockingRecv error: recv got EOF from 127.0.0.1:35312 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
W20260929 18:02:40.122040 30430 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:02:40.120710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35322 (local address 127.7.118.61:40801)
0929 18:02:40.120911 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:40.120915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:40.120997 (+ 82us) server_negotiation.cc:407] Connection header received
0929 18:02:40.121126 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:40.121129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:40.121183 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:40.121290 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:02:40.121895 (+ 605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35322: BlockingRecv error: recv got EOF from 127.0.0.1:35322 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 18:02:40.371086 30518 catalog_manager.cc:1259] Loaded cluster ID: c6654507bd86423b94e3e5c4bbaaebe7
I20260929 18:02:40.371184 30518 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1299ee17a8ae457689438596dbb843a9: loading cluster ID for follower catalog manager: success
I20260929 18:02:40.373432 30518 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1299ee17a8ae457689438596dbb843a9: acquiring CA information for follower catalog manager: success
I20260929 18:02:40.375041 30518 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1299ee17a8ae457689438596dbb843a9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:02:40.395042 30545 catalog_manager.cc:1259] Loaded cluster ID: c6654507bd86423b94e3e5c4bbaaebe7
I20260929 18:02:40.395174 30545 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 41a8dc8657cb42cd80d0e47339f45909: loading cluster ID for follower catalog manager: success
I20260929 18:02:40.396857 30545 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 41a8dc8657cb42cd80d0e47339f45909: acquiring CA information for follower catalog manager: success
I20260929 18:02:40.398097 30545 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 41a8dc8657cb42cd80d0e47339f45909: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:02:40.640645 30631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:40.636283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35332 (local address 127.7.118.61:40801)
0929 18:02:40.636588 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:40.636592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:40.636608 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:02:40.636759 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:40.636763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:40.636889 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:40.636989 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:02:40.637283 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:40.637850 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:40.638777 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:40.639018 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:40.639113 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:40.639487 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:02:40.639986 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:02:40.640107 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:40.640336 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 18:02:40.640442 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":93,"threads_started":1}
I20260929 18:02:40.648398 30637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:40.644253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45348 (local address 127.7.118.60:42509)
0929 18:02:40.644488 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:40.644492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:40.644506 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:02:40.644600 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:40.644603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:40.644746 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:40.644857 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:02:40.645125 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:40.645688 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:40.646796 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:40.647068 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:40.647160 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:40.647514 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:02:40.648025 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:02:40.648129 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:40.648205 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 18:02:40.648268 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":83,"threads_started":1}
I20260929 18:02:40.656534 30637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:40.652170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45354 (local address 127.7.118.60:42509)
0929 18:02:40.652393 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:40.652398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:40.652573 (+ 175us) server_negotiation.cc:407] Connection header received
0929 18:02:40.652745 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:40.652748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:40.652804 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:40.652881 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:02:40.653241 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:40.653996 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:40.655373 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:40.655627 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:40.655707 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:40.656044 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:02:40.656142 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:02:40.656227 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:40.656321 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 18:02:40.656381 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 18:02:40.656911 30631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:40.651922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35340 (local address 127.7.118.61:40801)
0929 18:02:40.652065 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:40.652072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:40.652475 (+ 403us) server_negotiation.cc:407] Connection header received
0929 18:02:40.652712 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:40.652715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:40.652782 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:40.652869 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:02:40.653113 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:40.653939 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:40.655020 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:40.655270 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:40.655389 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:40.656412 (+ 1023us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:02:40.656500 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:02:40.656547 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:40.656710 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 18:02:40.656777 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:02:40.657790 30643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:02:40.651995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54914 (local address 127.7.118.62:42283)
0929 18:02:40.652417 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0929 18:02:40.652424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:02:40.652708 (+ 284us) server_negotiation.cc:407] Connection header received
0929 18:02:40.652814 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:02:40.652819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:02:40.652995 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:02:40.653110 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:02:40.653438 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:40.653987 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:40.655399 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:02:40.655646 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:02:40.655777 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:02:40.656491 (+ 714us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:02:40.657183 (+ 692us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:02:40.657272 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 18:02:40.657570 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 18:02:40.657641 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":73,"threads_started":1}
I20260929 18:02:40.670274 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 30282
I20260929 18:02:40.677146 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 30357
I20260929 18:02:40.683554 7640 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8_cVK5/build/debug/bin/kudu with pid 30431
2026-09-29T18:02:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3336 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6576 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (303109 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST